Text.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Processing with Python\n",
    "\n",
    "Packages Discussued:\n",
    "\n",
    "- [readability-lxml](https://github.com/buriy/python-readability) and [BeautifulSoup](http://www.crummy.com/software/BeautifulSoup/)\n",
    "- [Pattern](http://www.clips.ua.ac.be/pattern)\n",
    "- [NLTK](http://www.nltk.org/)\n",
    "- [TextBlob](http://textblob.readthedocs.org/en/dev/)\n",
    "- [spaCy](https://honnibal.github.io/spaCy/updates.html)\n",
    "- [gensim](https://radimrehurek.com/gensim/)\n",
    "\n",
    "Other packages:\n",
    "\n",
    "- [MITIE](https://github.com/mit-nlp/MITIE)\n",
    "\n",
    "## NLP in Context\n",
    "\n",
    "> The science that has been developed around the facts of language passed through three stages before finding its true and unique object. First something called \"grammar\" was studied. This study, initiated by the Greeks and continued mainly by the French, was based on logic. It lacked a scientific approach and was detached from language itself. Its only aim was to give rules for distinguishing between correct and incorrect forms; it was a normative discipline, far removed from actual observation, and its scope was limited.\n",
    ">\n",
    "> &mdash; Ferdinand de Saussure\n",
    "\n",
    "### The State of the Art\n",
    "\n",
    "- Academic design for use alongside intelligent agents (AI discipline)\n",
    "- Relies on formal models or representations of knowledge & language\n",
    "- Models are adapted and augment through probabilistic methods and machine learning.\n",
    "- A small number of algorithms comprise the standard framework.\n",
    "\n",
    "Required:\n",
    "\n",
    "- Domain Knowledge\n",
    "- A Corpus in the Domain\n",
    "- Methods\n",
    "\n",
    "### The Data Science Pipeline\n",
    "\n",
    "![The Data Science Pipeline](figures/data_science_pipeline.png)\n",
    "\n",
    "### The NLP Pipeline\n",
    "\n",
    "![The NLP Pipeline](figures/nlp_pipeline.png)\n",
    "\n",
    "#### Morphology\n",
    "\n",
    "The study of the forms of things, words in particular.\n",
    "\n",
    "Consider pluralization for English:\n",
    "\n",
    "- Orthographic Rules: puppy → puppies\n",
    "- Morphological Rules: goose → geese or fish\n",
    "\n",
    "Major parsing tasks: \n",
    "\n",
    "- stemming\n",
    "- lemmatization\n",
    "- tokenization.\n",
    "\n",
    "#### Syntax\n",
    "\n",
    "The study of the rules for the formation of sentences.\n",
    "\n",
    "Major tasks:\n",
    "\n",
    "- chunking\n",
    "- parsing\n",
    "- feature parsing\n",
    "- grammars\n",
    "- NGram Models (perplexity)\n",
    "- Language generation\n",
    "\n",
    "#### Semantics\n",
    "\n",
    "The study of meaning.\n",
    "\n",
    "    I see what I eat.\n",
    "    I eat what I see.\n",
    "    He poached salmon.\n",
    "\n",
    "Major Tasks\n",
    "\n",
    "- Frame extraction\n",
    "- creation of TMRs\n",
    "- Question and answer systems\n",
    "\n",
    "#### Machine Learning\n",
    "\n",
    "Solve **Clustering Problems**:\n",
    "\n",
    "- Topic Modeling\n",
    "- Language Similarity\n",
    "- Document Association (authorship)\n",
    "\n",
    "Solve **Classification Problems**:\n",
    "\n",
    "- Language Detection\n",
    "- Sentiment Analysis\n",
    "- Part of Speech Tagging\n",
    "- Statistical Parsing\n",
    "- Much more\n",
    "\n",
    "Use of word _vectors_ to implement distance based metrics. \n",
    "\n",
    "## Setup and Dataset\n",
    "\n",
    "To install the required packages (hopefully to a virtual environment) you can download the `requirements.txt` and run:\n",
    "\n",
    "    $ pip install -r requirements.txt\n",
    "\n",
    "Or you can pip install each dependency as you need them.\n",
    "\n",
    "### Corpus Organization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing HTML and XML Documents to Text\n",
    "\n",
    "Much of the text that we're interested in is available on the web and formatted either as HTML or XML. It's not just web pages, however. Most eReader formats like ePub and Mobi are actually zip files containing XHTML. These semi-structured documents contain a lot of information, usually structural in nature. However, we want to get to the main body of the content of what we're looking for, disregarding other content that might be included such as headers for navigation, sidebars, ads and other extraneous content.\n",
    "\n",
    "On the web, there are several services that provide web pages in a \"readable\" fashion like [Instapaper](https://www.instapaper.com/) and [Clearly](https://evernote.com/clearly/). Some browsers might even come with a clutter and distraction free \"reading mode\" that seems to give us exactly the content that we're looking for. An option that I've used in the past is to either programmatically access these renderers, Instapaper even provides an [API](https://www.instapaper.com/api). However, for large corpora, we need to quickly and repeatably perform extraction, while maintaining the original documents. \n",
    "\n",
    "> Corpus management requires that the original documents be stored alongside preprocessed documents - do not make changes to the originals in place! See discussions of data lakes and data pipelines for more on ingesting to WORM storages.\n",
    "\n",
    "In Python, the fastest way to process HTML and XML text is with the [`lxml`](http://lxml.de/) library - a superfast XML parser that binds the C libraries `libxml2` and `libxslt`. However, the API for using `lxml` is a bit tricky, so instead use friendlier wrappers, `readability-lxml` and `BeautifulSoup`. \n",
    "\n",
    "For example, consider the following code to fetch an HTML web article from The Washington Post:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import codecs\n",
    "import requests\n",
    "\n",
    "from urlparse import urljoin\n",
    "from contextlib import closing\n",
    "\n",
    "chunk_size = 10**6  # Download 1 MB at a time.\n",
    "wpurl = \"http://wpo.st/\"  # Washington Post provides short links\n",
    "\n",
    "def fetch_webpage(url, path):\n",
    "    # Open up a stream request (to download large documents)\n",
    "    # Ensure that we will close when complete using contextlib\n",
    "    with closing(requests.get(url, stream=True)) as response:\n",
    "\n",
    "        # Check that the response was successful\n",
    "        if response.status_code == 200:\n",
    "            \n",
    "            # Write each chunk to disk with the correct encoding\n",
    "            with codecs.open(path, 'w', response.encoding) as f:\n",
    "                for chunk in response.iter_content(chunk_size,  decode_unicode=True):\n",
    "                    f.write(chunk)\n",
    "\n",
    "def fetch_wp_article(article_id):\n",
    "    path = \"%s.html\" % article_id\n",
    "    url  = urljoin(wpurl, article_id)\n",
    "    return fetch_webpage(url, path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fetch_webpage(\"http://www.koreadaily.com/news/read.asp?art_id=3283896\", \"korean.html\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "fetch_wp_article(\"nrRB0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "fetch_wp_article(\"uyRB0\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BeautifulSoup` allows us to search the DOM to extract particular elements, for example to load our document and find all the `<p>` tags, we would do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<p class=\"category-desc\"> The inside track on Washington politics. </p>\n",
      "<p class=\"invalid-email\">*Invalid email address</p>\n",
      "<p class=\"category-desc\"> The inside track on Washington politics. </p>\n",
      "<p class=\"invalid-email\">*Invalid email address</p>\n",
      "<p>Sign in or create an account so we can save this story to your Reading List. You'll be able to access the story from your Reading List on any computer, tablet or smartphone.</p>\n",
      "<p class=\"top-header-message\">Sign in to your account to save this article.</p>\n",
      "<p id=\"U9001274173114EdC\"></p>\n",
      "<p id=\"U1000696839467p6H\"> <i>It’s lowbrow. It’s messy. It could never be accused of being healthful. But we’d never let those formalities get between us and an order of crispy, crackly, delicious fried chicken. Whether it comes in a bucket or on a bun, or you eat it with your fingers or chopsticks, there’s a surprising variety to the Washington area’s fried chicken offerings. Here are some of the most irresistible.</i> </p>\n",
      "<p id=\"U1000696839467ntG\"></p>\n",
      "<p id=\"U9001274173114AOB\">Forget the cronut. Our newest favorite food chimera is the “rotissi-fried” chicken at <b>the Partisan</b>. Credit goes to chef Nate Anda, who dreamed up the dish: After a 12-hour brine, the chicken is rotisseried for two hours and then fried for two and a half minutes. Why both? “Everything is better once it’s fried in beef fat,” Anda said. We have to agree. Whether white or dark, the meat is succulent throughout. The batter-free frying leaves the simply seasoned skin rendered perfectly crisp, golden and translucent — cracklings, essentially. The sound of it shattering under the knife was music to our ears. And as if the lily needed further gilding, the chicken comes with a generous pour of honey hot sauce. The sauce is hard to resist, but try to reserve a few bites of unadorned chicken so you can fully appreciate this happy marriage of classic preparations.</p>\n",
      "<p id=\"U9001274173114XlF\"> <i>The Partisan, 709 D St. NW. 202-524-5322. <a class=\"showlink\" href=\"http://www.thepartisandc.com\">www.thepartisandc.com</a>. </i> </p>\n",
      "<p><strong>— Becky Krystal</strong></p>\n",
      "<p id=\"U9001274173114AdG\">When <a href=\"http://www.washingtonpost.com/lifestyle/food/bryan-voltaggio-from-a-teenager-amok-to-top-chef-masters/2013/07/22/682d35f8-ef04-11e2-bed3-b9b6fe264871_story.html\">Bryan Voltaggio</a> started planning the menu for <b>Family Meal</b>, his modern, upscale spin on a diner, his thoughts turned to home and the carryout meal he most enjoyed as a kid: fried chicken. “It was one of our favorite things,” he says. “It just seems like a family dinner.” And if he was creating a restaurant called Family Meal, fried chicken “had to be an important part of it.” But Voltaggio wanted to do it right, and set about testing cooking methods, brines, breadings and fryers. That long process paid off with a home run of a fried chicken dish that’s become the most popular item on Family Meal’s menu. The whole chickens spend 12 hours in a brine of pickle juice and roasted poultry stock before getting dredged, rested and dredged again in a mixture of flour, cornmeal and corn starch. After a dip in the top-of-the-line pressure fryer, the thighs, legs and breasts emerge with a crisp, salty skin that cracks open to reveal wonderfully warm and moist flesh. You don’t even need to dunk it in the house-made hot sauce that accompanies the dish, but really, who can resist? </p>\n",
      "<p id=\"U9001274173114THE\"> <i>Family Meal Ashburn, 20462 Exchange St. 703-726-9800; Frederick, 880 N. East St. 301-378-2895; Baltimore, 621 E. Pratt St. 410-601-3242. <a class=\"showlink\" href=\"http://www.voltfamilymeal.com\">www.voltfamilymeal.com</a>. </i> </p>\n",
      "<p><strong>— John Taylor</strong></p>\n",
      "<p id=\"U90012741731140AH\">Although it’s commonly served in Japan at karaoke bars, convenience stores and on street carts, kara age chicken — like most of the country’s food — is held to an extremely high standard. “It’s taken it to the nth degree of obsession and detail,” says Cizuka Seki, who, with her father Hiroshi, owns <b>Izakaya Seki</b> on V Street NW. “Kara age” is used to describe the method for deep-frying bite-size pieces of fish and, more commonly, chicken. Though there are subtle variations on the ubiquitous dish, most recipes call for chicken thighs marinated in soy sauce, coated in flour or corn starch and deep-fried in oil. Izakaya Seki’s version sticks closely to the formula. Probably. “I’m not even quite sure what my dad puts into it, because we don’t have recipes,” Seki says, though she’s certain wheat flour is involved. The result is a thin, tender coating that’s slightly softer than tempura. The accompanying ponzu sauce lends a tartness to the nubs.</p>\n",
      "<p id=\"U900127417311410F\"> <i>Izakaya Seki, 1117 V St. NW. 202-588-5841. <a class=\"showlink\" href=\"http://www.sekidc.com\">www.sekidc.com</a>. </i> </p>\n",
      "<p><strong>— Holley Simmons</strong></p>\n",
      "<p id=\"U9001274173114ewH\">Don’t waste your kimchi-stinking breath asking for more sauce at <b>BonChon</b>. The <a href=\"http://www.washingtonpost.com/goingoutguide/the-20-diner-the-zen-of-bonchon-chicken/2013/07/31/b879060c-f582-11e2-aa2e-4088616498b4_story.html\">South Korean fried chicken chain</a>, founded in 2002, is so dedicated to consistency that it doesn’t allow for any modifications. And why would you want to change anything, really? The made-to-order wings, drumsticks and strips are fried twice, resulting in a paper-thin crust that yields the same satisfying crack as shattering crème brulee with a spoon. Founder Jinduk Seh spent two years perfecting his secret sauces, which come in three flavors — soy garlic, hot and a blend of the two — and are brushed on by hand post-fry, piece by piece. True to BonChon’s commitment to uniformity, sauces are made exclusively in South Korea and distributed to all 140-plus BonChon locations, which means the wings you’re chewing on in Arlington are slathered with the same exact stuff as those in the Philippines. Joints like these are so common throughout Korea they’re called “chimeks,” which is a hybrid term that combines “chicken” with the Korean word for beer. Washington should be happy to have 10 BonChons within driving distance, plus a brand new Metro-accessible location near the Navy Yard.</p>\n",
      "<p id=\"U9001274173114bBI\"> <i>BonChon, 1015 Half St. SE and nine other locations in Maryland and Virginia. <a class=\"showlink\" href=\"http://www.bonchon.com\">www.bonchon.com</a>. </i> </p>\n",
      "<p><strong>— Holley Simmons</strong></p>\n",
      "<p id=\"U9001274173114NgD\">There’s not much agreement on what constitutes Maryland fried chicken. Some say it’s just a fresh Maryland chicken that’s pan-fried; others say it should be topped with white gravy, almost like a chicken-fried steak. The pan-fried chicken platter at <b> <br/>Crisfield Seafood</b> is a perfect example of the former style. Half of a chicken is dredged in flour, dusted with salt and pepper, and fried in a cast-iron pan. This preparation lends a snap and crunch to the exterior, and while the meat falls off the bone, the well-seasoned breading holds on. (The chicken is available only Friday through Sunday, and frequently sells out.) The Chesapeake fried chicken at <b>Hank’s Oyster Bar</b> in Dupont Circle and Capitol Hill is plumper than Crisfield’s version and seasoned with Old Bay, black pepper and cayenne, but the breading is softer and less crispy. It’s brined for 24 hours and deep-fried, rather than pan-fried, and it’s served only on Sunday. </p>\n",
      "<p id=\"U9001274173114wEE\"> <i>Crisfield Seafood, 8012 Georgia Ave., Silver Spring. 301-589-1306. <a class=\"showlink\" href=\"http://www.crisfieldseafood.com\">www.crisfieldseafood.com</a>. Hank’s Oyster Bar, 1624 Q St. NW. 202-462-4265; 633 Pennsylvania Ave. SE. 202-733-1971. <a class=\"showlink\" href=\"http://www.hanksoysterbar.com\">www.hanksoysterbar.com</a>.</i> </p>\n",
      "<p><strong>— Fritz Hahn</strong></p>\n",
      "<p id=\"U9001274173114bsG\">Self-consciousness may prevent you from ordering fried chicken in a white-tablecloth restaurant. It feels incongruous — gauche, almost — to dig into picnic fare at the kind of place where you should be ordering risotto or tartare or something that comes with mousse, gelee or foam. But you have to override that adult voice in the back of your head, because if you don’t, you’ll miss out on <b>Central Michel Richard</b>’s famed fried chicken plate ($24 at lunch, $25 for dinner), which remains as good as ever. Though it’s no longer sold by the bucket to go, Michel Richard’s KFC-inspired crispy breast and thigh come stacked atop a pool of the butteriest mashed potatoes you’ll ever taste. It’s such a dignified presentation that this most American of dishes almost could pass as (gasp!) French. Self-consciousness should, however, steer you toward using a knife and fork — not your fingers — to eat the chicken. It is, after all, that kind of a place. </p>\n",
      "<p id=\"U9001274173114p0D\"> <i>Central Michel Richard, 1001 Pennsylvania Ave. NW. 202-626-0015. <a class=\"showlink\" href=\"http://www.centralmichelrichard.com\">www.centralmichelrichard.com</a> <i>. </i> </i> </p>\n",
      "<p></p>\n",
      "<p><strong>— Maura Judkis</strong></p>\n",
      "<p id=\"U900127417311467G\"></p>\n",
      "<p id=\"U9001274173114o0G\">If you believe the lore, Nashville hot chicken was basically a crime of passion, created as a blistering rebuke to a no-account Romeo who couldn’t keep his hands off other women. Alas, this wolf was also a chili head who found pleasure, not pain, in this dish of revenge served hot. Decades later, chefs are latching onto this addictive form of punishment. Aaron Silverman served an ultra-refined version at Rose’s Luxury for months, and now Eric Reid, chef and co-owner of <b>Reserve 2216</b> in Del Ray, has developed his own take on hot chicken, even if he’s never actually enjoyed it in Nashville. He marinates an airline cut (boneless breast with the drumette wing attached) in buttermilk and Crystal hot sauce before dredging the chicken in seasoned flour and frying it. Reid ditches the traditional white bread base in favor of collards and a side of corn bread waffles. He finishes the dish with a combination of Cajun seasonings and more Crystal hot sauce for a moist, crispy bird that bites back. But not too hard. This is Alexandria, after all. </p>\n",
      "<p id=\"U900127417311481\"> <i>Reserve 2216, 2216 Mount Vernon Ave., Alexandria. 703-549-2889. <a class=\"showlink\" href=\"http://www.drpreserve.com\">www.drpreserve.com</a>.</i> </p>\n",
      "<p><strong>— Tim Carman</strong></p>\n",
      "<p id=\"U9001274173114HvD\"></p>\n",
      "<p id=\"U90012741731145tH\">The sole virtue of most fast-food operations is consistency. Whether you bite into a Big Mac in Bethesda or Beijing, the sandwich should taste the same. The menu at <b>Popeyes</b> follows suit, but it deviates from the brand-name competition in an important respect: The signature at Popeyes could pass for home cooking (well, if your home had a vat of clean, hot oil and a person with a Southern accent tending the meal). Maybe that accounts for my occasional forays to the chicken fryer after a bum restaurant-review excursion. No matter where I eat my order, inevitably “spicy,” I know I can count on a coating that smacks of cayenne, paprika and even crushed cornflakes, and chicken that spurts with juice. The shatter is audible; the golden crumbs fly everywhere, but end up on my tongue. No one-trick pony, Popeyes has hot and tender buttermilk biscuits that bolster my favorite excuse to snack low on the food chain. Once, I got home to discover a clerk had forgotten to pack bread in my bag. I almost cried. Instead, I consoled myself with another piece of chicken. </p>\n",
      "<p id=\"U9001274173114EOF\"> <i>Popeyes has locations throughout the D.C. metro area. <a class=\"showlink\" href=\"http://www.popeyes.com\">www.popeyes.com</a>.</i> </p>\n",
      "<p><strong>— Tom Sietsema</strong></p>\n",
      "<p id=\"U9001274173114JiB\"></p>\n",
      "<p id=\"U9001274173114mFG\">The fried chicken sandwich hasn’t been the same since KFC’s Double Down turned a guilty pleasure into an outright farce, using crispy white-meat fillets as both the sandwich’s primary protein and the oily handles by which you eat the monstrosity. Leave it to Rob Sonderman, pitmaster and co-owner of <b>DCity Smokehouse</b>, to bring dignity back to the bite. His Den-Den — named for co-creator and pitmaster-in-training <br/>Dennis Geddie — begins with boneless thighs marinated in buttermilk, hot sauce and honey. Sonderman then dredges the meat in seasoned flour before dropping the thighs into the fryer. Generously stuffed into a grilled hoagie roll with lettuce, tomato and crispy onions, the chicken is finished with two sauces, including a house-made cilantro ranch. Technically, the Den-Den ($9.25) is one of the few smokeless items on DCity’s menu (unless you count the chipotle peppers in the hot sauce). No matter. You won’t care the minute you sink your teeth into that heaping hoagie of spicy thigh meat.</p>\n",
      "<p id=\"U9001274173114vPC\"> <i>DCity Smokehouse, 8 Florida Ave. NW. 202-733-1919. <a class=\"showlink\" href=\"http://www.dcitysmokehouse.com\">www.dcitysmokehouse.com</a>. </i> </p>\n",
      "<p><strong>— Tim Carman</strong></p>\n",
      "<p id=\"U9001274173114L3B\"></p>\n",
      "<p id=\"U9001274173114yyH\">Hearty is the appetite that can handle <b>Oohh’s and Aahh’s</b> chef-owner Oji Abbott’s boneless fried chicken breast without taking home leftovers. He buys local — from Hartman Meat Co. in Northeast Washington — and butterflies each 14-ounce portion ($12.95), which results in a lot of real estate for the crispy, well-seasoned coating. Abbott chalks up the consistently moist meat to proper cooking time and temperature, and to the recipe he learned from his grandmother.</p>\n",
      "<p id=\"U9001274173114gCF\"> <i>Oohh’s and Aahh’s, 1005 U St. NW. 202-667-7142. <a class=\"showlink\" href=\"http://www.oohhsnaahhs.com\">www.oohhsnaahhs.com</a>.</i> </p>\n",
      "<p><strong>— Bonnie S. Benwick</strong></p>\n",
      "<p id=\"U9001274173114ZvF\"></p>\n",
      "<p id=\"U9001274173114wdH\">It’s all too easy to chomp through an order of Boardwalk Chicken at the shore-happy <b>Pop’s Sea Bar</b> in Adams Morgan. The bite-size pieces of dark-meat-only bird are brined for two hours, then treated to a buttermilk bath until they are coated with plain flour and flash-fried to order. A generous hand with salt and pepper just before serving means the effect of that seasoning builds as you empty the single-serving basket ($8.99). Ask for two portions of the accompanying Jersey sauce so you’ll have enough of its horseradish-y, kitchen-sink blend for every bite.</p>\n",
      "<p id=\"U9001274173114ycC\"> <i>Pop’s Sea Bar, 1817 Columbia Rd. NW. 202-534-3933. <a class=\"showlink\" href=\"http://www.popsseabar.com\">www.popsseabar.com</a>.</i> </p>\n",
      "<p><strong>— Bonnie S. Benwick</strong></p>\n",
      "<p id=\"U9001274173114ZfD\"></p>\n",
      "<p id=\"U9001274173114RpD\">Chicken tenders are often relegated to the children’s menu, but the chicken Tendies at <b>GBD</b> are a fine meal for adults and children alike. Each white-meat tender comes with a dark, crispy outer layer with a heavy dose of salt and spice. But the best part about chicken tenders is the dipping, and GBD offers nine sauces, including a take on <a href=\"http://www.washingtonpost.com/lifestyle/food/mumbo-sauce-gets-gentrified/2013/07/08/b1011ade-cc67-11e2-9f1a-1a7cdee20287_story.html\">D.C.’s own mumbo sauce</a>, buttermilk ranch, chipotle barbecue and Frankenbutter, which combines Frank’s RedHot sauce with butter. Ask for the $5.50 Saucetown option to try all nine. </p>\n",
      "<p id=\"U9001274173114bCE\"> <i>GBD, 1323 Connecticut Ave. NW. 202-524-5210. <a class=\"showlink\" href=\"http://www.gbdchickendoughnuts.com/\">www.gbdchickendoughnuts.com</a>.</i> </p>\n",
      "<p><strong>— Margaret Ely</strong></p>\n",
      "<p id=\"U9001274173114xGE\"></p>\n",
      "<p id=\"U90012741731149NI\">Fried chicken fans can argue the merits of white meat vs. dark meat, or whether it’s better to chow down on a drumstick or the breast. But one thing we all can agree on is that the outer layer — the breading and the skin — is the most important element of a memorable piece of fried chicken. And sometimes you just want to savor the flavor of the skin — deeply spiced, perfect crunch — without filling up on meat or having to deal with bones. And that is when you grab one of the bar stools at R.J. Cooper’s <b>Gypsy Soul</b> in Fairfax’s Mosaic District. Cooper’s chicken skins ($9) are twisted slivers and shards, decadently salty and crackling with paprika, cayenne pepper and garlic. The dish arrives with a house-made “roof top honey-snake oil,” but it’s best to let these beauties shine on their own. </p>\n",
      "<p id=\"U9001274173114vUB\"> <i>Gypsy Soul, 8296 Glass Alley, Fairfax. 703-992-0933. <a class=\"showlink\" href=\"http://www.gypsysoul-va.com\">www.gypsysoul-va.com</a>.</i> </p>\n",
      "<p><strong>— Fritz Hahn</strong></p>\n",
      "<p class=\"section-instream\">goingoutguide</p>\n",
      "<p class=\"subsection-instream\"></p>\n",
      "<p class=\"blogname-instream\"></p>\n",
      "<p class=\"headline\"></p>\n",
      "<p class=\"tagline\"></p>\n",
      "<p class=\"error hide\">Please provide a valid email address. </p>\n",
      "<p class=\"headline\">The Freddie Gray case </p>\n",
      "<p class=\"follow-tagline follow\"> Sign up for email updates on the trials.</p>\n",
      "<p class=\"follow-tagline unfollow hide\">You’ve signed up for email updates on this story.</p>\n",
      "<p class=\"error col-xs-12\">Please provide a valid email address. </p>\n",
      "<p class=\"headline\"> <span class=\"campaign-txt\"> <span class=\"bold\">Campaign 2016 </span> <span class=\"flower-image\"> </span> </span> Email Updates </p>\n",
      "<p class=\"follow-tagline follow\">Get the best analysis of the presidential race.</p>\n",
      "<p class=\"follow-tagline unfollow hide\">You’ve signed up for email updates on this story.</p>\n",
      "<p class=\"error col-xs-12\">Please provide a valid email address. </p>\n",
      "<p class=\"headline\">Get Zika news by email</p>\n",
      "<p class=\"follow-tagline follow\"> We will update you when news breaks about the virus.</p>\n",
      "<p class=\"follow-tagline unfollow hide\">You’ve signed up for email updates on this story.</p>\n",
      "<p class=\"error col-xs-12\">Please provide a valid email address. </p>\n",
      "<p class=\"title\">SuperFan Badge</p>\n",
      "<p>SuperFan badge holders consistently post smart, timely comments about Washington area sports and teams.</p>\n",
      "<p class=\"echo-badge-info-link\"><a href=\"//www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">More about badges</a> | <a href=\"http://www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">Request a badge</a></p>\n",
      "<p class=\"title\">Culture Connoisseur Badge</p>\n",
      "<p>Culture Connoisseurs consistently offer thought-provoking, timely comments on the arts, lifestyle and entertainment.</p>\n",
      "<p class=\"echo-badge-info-link\"><a href=\"//www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">More about badges</a> | <a href=\"http://www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">Request a badge</a></p>\n",
      "<p class=\"title\">Fact Checker Badge</p>\n",
      "<p>Fact Checkers contribute questions, information and facts to <a href=\"//www.washingtonpost.com/blogs/fact-checker\" target=\"_badgeinfo\">The Fact Checker</a>.</p>\n",
      "<p class=\"echo-badge-info-link\"><a href=\"//www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">More about badges</a> | <a href=\"http://www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">Request a badge</a></p>\n",
      "<p class=\"title\">Washingtologist Badge</p>\n",
      "<p>Washingtologists consistently post thought-provoking, timely comments on events, communities, and trends in the Washington area.</p>\n",
      "<p class=\"echo-badge-info-link\"><a href=\"//www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">More about badges</a> | <a href=\"http://www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">Request a badge</a></p>\n",
      "<p class=\"title\">Post Writer Badge</p>\n",
      "<p>This commenter is a Washington Post editor, reporter or producer.</p>\n",
      "<p class=\"title\">Post Forum Badge</p>\n",
      "<p>Post Forum members consistently offer thought-provoking, timely comments on politics, national and international affairs.</p>\n",
      "<p class=\"echo-badge-info-link\"><a href=\"//www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">More about badges</a> | <a href=\"http://www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">Request a badge</a></p>\n",
      "<p class=\"title\">Weather Watcher Badge</p>\n",
      "<p>Weather Watchers consistently offer thought-provoking, timely comments on climates and forecasts.</p>\n",
      "<p class=\"echo-badge-info-link\"><a href=\"//www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">More about badges</a> | <a href=\"http://www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">Request a badge</a></p>\n",
      "<p class=\"title\">World Watcher Badge</p>\n",
      "<p>World Watchers consistently offer thought-provoking, timely comments on international affairs.</p>\n",
      "<p class=\"echo-badge-info-link\"><a href=\"//www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">More about badges</a> | <a href=\"http://www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">Request a badge</a></p>\n",
      "<p class=\"title\">Post Contributor Badge</p>\n",
      "<p>This commenter is a Washington Post contributor. Post contributors aren’t staff, but may write articles or columns. In some cases, contributors are sources or experts quoted in a story.</p>\n",
      "<p class=\"echo-badge-info-link\"><a href=\"//www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">More about badges</a> | <a href=\"http://www.washingtonpost.com/wp-srv/interactivity/get-a-badge.html\" target=\"_badgeinfo\">Request a badge</a></p>\n",
      "<p class=\"title\">Post Recommended</p>\n",
      "<p>Washington Post reporters or editors recommend this comment or reader post.</p>\n",
      "<p>You must be logged in to report a comment.</p>\n",
      "<p>You must be logged in to recommend a comment.</p>\n",
      "<p>Comments our editors find particularly useful or relevant are displayed in <strong>Top Comments</strong>, as are comments by users with these badges: <strong><span class=\"badge-list\"></span></strong>. Replies to those posts appear here, as well as posts by staff writers.</p>\n",
      "<p>All comments are posted in the <strong>All Comments</strong> tab.</p>\n",
      "<p>To pause and restart automatic updates, click \"Live\" or \"Paused\". If paused, you'll be notified of the number of additional comments that have come in.</p>\n",
      "<p>Play right from this page</p>\n",
      "<p> It's 3D Mahjongg- you don't even need to wear 3D glasses! </p>\n",
      "<p> Online crossword. </p>\n",
      "<p> Spider Solitaire is known as the king of all solitaire games! </p>\n",
      "<p> Challenge your crossword skills everyday with a huge variety of puzzles waiting for you to solve. </p>\n",
      "<p id=\"newsletter-section\">goingoutguide</p>\n",
      "<p id=\"newsletter-subsection\"></p>\n",
      "<p id=\"newsletter-blogname\"></p>\n",
      "<p class=\"headline\" id=\"newsletter-headline\"><i class=\"fa fa-check\" id=\"headline-checked\"></i></p>\n",
      "<p class=\"title\" id=\"newsletter-tagline\"></p>\n",
      "<p class=\"title\" id=\"subscribed-confirmation\"><span>Success!</span> Check your inbox for details.</p>\n",
      "<p class=\"newsLetter-error-msg\"> Please enter a valid email address </p>\n",
      "<p class=\"title\">You might also like: </p>\n",
      "<p class=\"title \"></p>\n",
      "<p class=\"title \\\"></p>\n",
      "<p class=\"title\"></p>\n",
      "<p class=\"title\" id=\"all-newsletters-lbl\"><a href=\"https://subscribe.washingtonpost.com/newsletters\">See all newsletters</a></p>\n"
     ]
    }
   ],
   "source": [
    "import bs4\n",
    "\n",
    "def get_soup(path):\n",
    "    with open(path, 'r') as f:\n",
    "        return bs4.BeautifulSoup(f, \"lxml\") # Note the use of the lxml parser\n",
    "\n",
    "for p in get_soup(\"nrRB0.html\").find_all('p'):\n",
    "    print p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to print out only the text with no nodes, do the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " The inside track on Washington politics. \n",
      "\n",
      "*Invalid email address\n",
      "\n",
      " The inside track on Washington politics. \n",
      "\n",
      "*Invalid email address\n",
      "\n",
      "Sign in or create an account so we can save this story to your Reading List. You'll be able to access the story from your Reading List on any computer, tablet or smartphone.\n",
      "\n",
      "Sign in to your account to save this article.\n",
      "\n",
      "\n",
      "\n",
      " It’s lowbrow. It’s messy. It could never be accused of being healthful. But we’d never let those formalities get between us and an order of crispy, crackly, delicious fried chicken. Whether it comes in a bucket or on a bun, or you eat it with your fingers or chopsticks, there’s a surprising variety to the Washington area’s fried chicken offerings. Here are some of the most irresistible. \n",
      "\n",
      "\n",
      "\n",
      "Forget the cronut. Our newest favorite food chimera is the “rotissi-fried” chicken at the Partisan. Credit goes to chef Nate Anda, who dreamed up the dish: After a 12-hour brine, the chicken is rotisseried for two hours and then fried for two and a half minutes. Why both? “Everything is better once it’s fried in beef fat,” Anda said. We have to agree. Whether white or dark, the meat is succulent throughout. The batter-free frying leaves the simply seasoned skin rendered perfectly crisp, golden and translucent — cracklings, essentially. The sound of it shattering under the knife was music to our ears. And as if the lily needed further gilding, the chicken comes with a generous pour of honey hot sauce. The sauce is hard to resist, but try to reserve a few bites of unadorned chicken so you can fully appreciate this happy marriage of classic preparations.\n",
      "\n",
      " The Partisan, 709 D St. NW. 202-524-5322. www.thepartisandc.com.  \n",
      "\n",
      "— Becky Krystal\n",
      "\n",
      "When Bryan Voltaggio started planning the menu for Family Meal, his modern, upscale spin on a diner, his thoughts turned to home and the carryout meal he most enjoyed as a kid: fried chicken. “It was one of our favorite things,” he says. “It just seems like a family dinner.” And if he was creating a restaurant called Family Meal, fried chicken “had to be an important part of it.” But Voltaggio wanted to do it right, and set about testing cooking methods, brines, breadings and fryers. That long process paid off with a home run of a fried chicken dish that’s become the most popular item on Family Meal’s menu. The whole chickens spend 12 hours in a brine of pickle juice and roasted poultry stock before getting dredged, rested and dredged again in a mixture of flour, cornmeal and corn starch. After a dip in the top-of-the-line pressure fryer, the thighs, legs and breasts emerge with a crisp, salty skin that cracks open to reveal wonderfully warm and moist flesh. You don’t even need to dunk it in the house-made hot sauce that accompanies the dish, but really, who can resist? \n",
      "\n",
      " Family Meal Ashburn, 20462 Exchange St. 703-726-9800; Frederick, 880 N. East St. 301-378-2895; Baltimore, 621 E. Pratt St. 410-601-3242. www.voltfamilymeal.com.  \n",
      "\n",
      "— John Taylor\n",
      "\n",
      "Although it’s commonly served in Japan at karaoke bars, convenience stores and on street carts, kara age chicken — like most of the country’s food — is held to an extremely high standard. “It’s taken it to the nth degree of obsession and detail,” says Cizuka Seki, who, with her father Hiroshi, owns Izakaya Seki on V Street NW. “Kara age” is used to describe the method for deep-frying bite-size pieces of fish and, more commonly, chicken. Though there are subtle variations on the ubiquitous dish, most recipes call for chicken thighs marinated in soy sauce, coated in flour or corn starch and deep-fried in oil. Izakaya Seki’s version sticks closely to the formula. Probably. “I’m not even quite sure what my dad puts into it, because we don’t have recipes,” Seki says, though she’s certain wheat flour is involved. The result is a thin, tender coating that’s slightly softer than tempura. The accompanying ponzu sauce lends a tartness to the nubs.\n",
      "\n",
      " Izakaya Seki, 1117 V St. NW. 202-588-5841. www.sekidc.com.  \n",
      "\n",
      "— Holley Simmons\n",
      "\n",
      "Don’t waste your kimchi-stinking breath asking for more sauce at BonChon. The South Korean fried chicken chain, founded in 2002, is so dedicated to consistency that it doesn’t allow for any modifications. And why would you want to change anything, really? The made-to-order wings, drumsticks and strips are fried twice, resulting in a paper-thin crust that yields the same satisfying crack as shattering crème brulee with a spoon. Founder Jinduk Seh spent two years perfecting his secret sauces, which come in three flavors — soy garlic, hot and a blend of the two — and are brushed on by hand post-fry, piece by piece. True to BonChon’s commitment to uniformity, sauces are made exclusively in South Korea and distributed to all 140-plus BonChon locations, which means the wings you’re chewing on in Arlington are slathered with the same exact stuff as those in the Philippines. Joints like these are so common throughout Korea they’re called “chimeks,” which is a hybrid term that combines “chicken” with the Korean word for beer. Washington should be happy to have 10 BonChons within driving distance, plus a brand new Metro-accessible location near the Navy Yard.\n",
      "\n",
      " BonChon, 1015 Half St. SE and nine other locations in Maryland and Virginia. www.bonchon.com.  \n",
      "\n",
      "— Holley Simmons\n",
      "\n",
      "There’s not much agreement on what constitutes Maryland fried chicken. Some say it’s just a fresh Maryland chicken that’s pan-fried; others say it should be topped with white gravy, almost like a chicken-fried steak. The pan-fried chicken platter at  Crisfield Seafood is a perfect example of the former style. Half of a chicken is dredged in flour, dusted with salt and pepper, and fried in a cast-iron pan. This preparation lends a snap and crunch to the exterior, and while the meat falls off the bone, the well-seasoned breading holds on. (The chicken is available only Friday through Sunday, and frequently sells out.) The Chesapeake fried chicken at Hank’s Oyster Bar in Dupont Circle and Capitol Hill is plumper than Crisfield’s version and seasoned with Old Bay, black pepper and cayenne, but the breading is softer and less crispy. It’s brined for 24 hours and deep-fried, rather than pan-fried, and it’s served only on Sunday. \n",
      "\n",
      " Crisfield Seafood, 8012 Georgia Ave., Silver Spring. 301-589-1306. www.crisfieldseafood.com. Hank’s Oyster Bar, 1624 Q St. NW. 202-462-4265; 633 Pennsylvania Ave. SE. 202-733-1971. www.hanksoysterbar.com. \n",
      "\n",
      "— Fritz Hahn\n",
      "\n",
      "Self-consciousness may prevent you from ordering fried chicken in a white-tablecloth restaurant. It feels incongruous — gauche, almost — to dig into picnic fare at the kind of place where you should be ordering risotto or tartare or something that comes with mousse, gelee or foam. But you have to override that adult voice in the back of your head, because if you don’t, you’ll miss out on Central Michel Richard’s famed fried chicken plate ($24 at lunch, $25 for dinner), which remains as good as ever. Though it’s no longer sold by the bucket to go, Michel Richard’s KFC-inspired crispy breast and thigh come stacked atop a pool of the butteriest mashed potatoes you’ll ever taste. It’s such a dignified presentation that this most American of dishes almost could pass as (gasp!) French. Self-consciousness should, however, steer you toward using a knife and fork — not your fingers — to eat the chicken. It is, after all, that kind of a place. \n",
      "\n",
      " Central Michel Richard, 1001 Pennsylvania Ave. NW. 202-626-0015. www.centralmichelrichard.com .   \n",
      "\n",
      "\n",
      "\n",
      "— Maura Judkis\n",
      "\n",
      "\n",
      "\n",
      "If you believe the lore, Nashville hot chicken was basically a crime of passion, created as a blistering rebuke to a no-account Romeo who couldn’t keep his hands off other women. Alas, this wolf was also a chili head who found pleasure, not pain, in this dish of revenge served hot. Decades later, chefs are latching onto this addictive form of punishment. Aaron Silverman served an ultra-refined version at Rose’s Luxury for months, and now Eric Reid, chef and co-owner of Reserve 2216 in Del Ray, has developed his own take on hot chicken, even if he’s never actually enjoyed it in Nashville. He marinates an airline cut (boneless breast with the drumette wing attached) in buttermilk and Crystal hot sauce before dredging the chicken in seasoned flour and frying it. Reid ditches the traditional white bread base in favor of collards and a side of corn bread waffles. He finishes the dish with a combination of Cajun seasonings and more Crystal hot sauce for a moist, crispy bird that bites back. But not too hard. This is Alexandria, after all. \n",
      "\n",
      " Reserve 2216, 2216 Mount Vernon Ave., Alexandria. 703-549-2889. www.drpreserve.com. \n",
      "\n",
      "— Tim Carman\n",
      "\n",
      "\n",
      "\n",
      "The sole virtue of most fast-food operations is consistency. Whether you bite into a Big Mac in Bethesda or Beijing, the sandwich should taste the same. The menu at Popeyes follows suit, but it deviates from the brand-name competition in an important respect: The signature at Popeyes could pass for home cooking (well, if your home had a vat of clean, hot oil and a person with a Southern accent tending the meal). Maybe that accounts for my occasional forays to the chicken fryer after a bum restaurant-review excursion. No matter where I eat my order, inevitably “spicy,” I know I can count on a coating that smacks of cayenne, paprika and even crushed cornflakes, and chicken that spurts with juice. The shatter is audible; the golden crumbs fly everywhere, but end up on my tongue. No one-trick pony, Popeyes has hot and tender buttermilk biscuits that bolster my favorite excuse to snack low on the food chain. Once, I got home to discover a clerk had forgotten to pack bread in my bag. I almost cried. Instead, I consoled myself with another piece of chicken. \n",
      "\n",
      " Popeyes has locations throughout the D.C. metro area. www.popeyes.com. \n",
      "\n",
      "— Tom Sietsema\n",
      "\n",
      "\n",
      "\n",
      "The fried chicken sandwich hasn’t been the same since KFC’s Double Down turned a guilty pleasure into an outright farce, using crispy white-meat fillets as both the sandwich’s primary protein and the oily handles by which you eat the monstrosity. Leave it to Rob Sonderman, pitmaster and co-owner of DCity Smokehouse, to bring dignity back to the bite. His Den-Den — named for co-creator and pitmaster-in-training Dennis Geddie — begins with boneless thighs marinated in buttermilk, hot sauce and honey. Sonderman then dredges the meat in seasoned flour before dropping the thighs into the fryer. Generously stuffed into a grilled hoagie roll with lettuce, tomato and crispy onions, the chicken is finished with two sauces, including a house-made cilantro ranch. Technically, the Den-Den ($9.25) is one of the few smokeless items on DCity’s menu (unless you count the chipotle peppers in the hot sauce). No matter. You won’t care the minute you sink your teeth into that heaping hoagie of spicy thigh meat.\n",
      "\n",
      " DCity Smokehouse, 8 Florida Ave. NW. 202-733-1919. www.dcitysmokehouse.com.  \n",
      "\n",
      "— Tim Carman\n",
      "\n",
      "\n",
      "\n",
      "Hearty is the appetite that can handle Oohh’s and Aahh’s chef-owner Oji Abbott’s boneless fried chicken breast without taking home leftovers. He buys local — from Hartman Meat Co. in Northeast Washington — and butterflies each 14-ounce portion ($12.95), which results in a lot of real estate for the crispy, well-seasoned coating. Abbott chalks up the consistently moist meat to proper cooking time and temperature, and to the recipe he learned from his grandmother.\n",
      "\n",
      " Oohh’s and Aahh’s, 1005 U St. NW. 202-667-7142. www.oohhsnaahhs.com. \n",
      "\n",
      "— Bonnie S. Benwick\n",
      "\n",
      "\n",
      "\n",
      "It’s all too easy to chomp through an order of Boardwalk Chicken at the shore-happy Pop’s Sea Bar in Adams Morgan. The bite-size pieces of dark-meat-only bird are brined for two hours, then treated to a buttermilk bath until they are coated with plain flour and flash-fried to order. A generous hand with salt and pepper just before serving means the effect of that seasoning builds as you empty the single-serving basket ($8.99). Ask for two portions of the accompanying Jersey sauce so you’ll have enough of its horseradish-y, kitchen-sink blend for every bite.\n",
      "\n",
      " Pop’s Sea Bar, 1817 Columbia Rd. NW. 202-534-3933. www.popsseabar.com. \n",
      "\n",
      "— Bonnie S. Benwick\n",
      "\n",
      "\n",
      "\n",
      "Chicken tenders are often relegated to the children’s menu, but the chicken Tendies at GBD are a fine meal for adults and children alike. Each white-meat tender comes with a dark, crispy outer layer with a heavy dose of salt and spice. But the best part about chicken tenders is the dipping, and GBD offers nine sauces, including a take on D.C.’s own mumbo sauce, buttermilk ranch, chipotle barbecue and Frankenbutter, which combines Frank’s RedHot sauce with butter. Ask for the $5.50 Saucetown option to try all nine. \n",
      "\n",
      " GBD, 1323 Connecticut Ave. NW. 202-524-5210. www.gbdchickendoughnuts.com. \n",
      "\n",
      "— Margaret Ely\n",
      "\n",
      "\n",
      "\n",
      "Fried chicken fans can argue the merits of white meat vs. dark meat, or whether it’s better to chow down on a drumstick or the breast. But one thing we all can agree on is that the outer layer — the breading and the skin — is the most important element of a memorable piece of fried chicken. And sometimes you just want to savor the flavor of the skin — deeply spiced, perfect crunch — without filling up on meat or having to deal with bones. And that is when you grab one of the bar stools at R.J. Cooper’s Gypsy Soul in Fairfax’s Mosaic District. Cooper’s chicken skins ($9) are twisted slivers and shards, decadently salty and crackling with paprika, cayenne pepper and garlic. The dish arrives with a house-made “roof top honey-snake oil,” but it’s best to let these beauties shine on their own. \n",
      "\n",
      " Gypsy Soul, 8296 Glass Alley, Fairfax. 703-992-0933. www.gypsysoul-va.com. \n",
      "\n",
      "— Fritz Hahn\n",
      "\n",
      "goingoutguide\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Please provide a valid email address. \n",
      "\n",
      "The Freddie Gray case \n",
      "\n",
      " Sign up for email updates on the trials.\n",
      "\n",
      "You’ve signed up for email updates on this story.\n",
      "\n",
      "Please provide a valid email address. \n",
      "\n",
      "  Campaign 2016     Email Updates \n",
      "\n",
      "Get the best analysis of the presidential race.\n",
      "\n",
      "You’ve signed up for email updates on this story.\n",
      "\n",
      "Please provide a valid email address. \n",
      "\n",
      "Get Zika news by email\n",
      "\n",
      " We will update you when news breaks about the virus.\n",
      "\n",
      "You’ve signed up for email updates on this story.\n",
      "\n",
      "Please provide a valid email address. \n",
      "\n",
      "SuperFan Badge\n",
      "\n",
      "SuperFan badge holders consistently post smart, timely comments about Washington area sports and teams.\n",
      "\n",
      "More about badges | Request a badge\n",
      "\n",
      "Culture Connoisseur Badge\n",
      "\n",
      "Culture Connoisseurs consistently offer thought-provoking, timely comments on the arts, lifestyle and entertainment.\n",
      "\n",
      "More about badges | Request a badge\n",
      "\n",
      "Fact Checker Badge\n",
      "\n",
      "Fact Checkers contribute questions, information and facts to The Fact Checker.\n",
      "\n",
      "More about badges | Request a badge\n",
      "\n",
      "Washingtologist Badge\n",
      "\n",
      "Washingtologists consistently post thought-provoking, timely comments on events, communities, and trends in the Washington area.\n",
      "\n",
      "More about badges | Request a badge\n",
      "\n",
      "Post Writer Badge\n",
      "\n",
      "This commenter is a Washington Post editor, reporter or producer.\n",
      "\n",
      "Post Forum Badge\n",
      "\n",
      "Post Forum members consistently offer thought-provoking, timely comments on politics, national and international affairs.\n",
      "\n",
      "More about badges | Request a badge\n",
      "\n",
      "Weather Watcher Badge\n",
      "\n",
      "Weather Watchers consistently offer thought-provoking, timely comments on climates and forecasts.\n",
      "\n",
      "More about badges | Request a badge\n",
      "\n",
      "World Watcher Badge\n",
      "\n",
      "World Watchers consistently offer thought-provoking, timely comments on international affairs.\n",
      "\n",
      "More about badges | Request a badge\n",
      "\n",
      "Post Contributor Badge\n",
      "\n",
      "This commenter is a Washington Post contributor. Post contributors aren’t staff, but may write articles or columns. In some cases, contributors are sources or experts quoted in a story.\n",
      "\n",
      "More about badges | Request a badge\n",
      "\n",
      "Post Recommended\n",
      "\n",
      "Washington Post reporters or editors recommend this comment or reader post.\n",
      "\n",
      "You must be logged in to report a comment.\n",
      "\n",
      "You must be logged in to recommend a comment.\n",
      "\n",
      "Comments our editors find particularly useful or relevant are displayed in Top Comments, as are comments by users with these badges: . Replies to those posts appear here, as well as posts by staff writers.\n",
      "\n",
      "All comments are posted in the All Comments tab.\n",
      "\n",
      "To pause and restart automatic updates, click \"Live\" or \"Paused\". If paused, you'll be notified of the number of additional comments that have come in.\n",
      "\n",
      "Play right from this page\n",
      "\n",
      " It's 3D Mahjongg- you don't even need to wear 3D glasses! \n",
      "\n",
      " Online crossword. \n",
      "\n",
      " Spider Solitaire is known as the king of all solitaire games! \n",
      "\n",
      " Challenge your crossword skills everyday with a huge variety of puzzles waiting for you to solve. \n",
      "\n",
      "goingoutguide\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Success! Check your inbox for details.\n",
      "\n",
      " Please enter a valid email address \n",
      "\n",
      "You might also like: \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "See all newsletters\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for p in get_soup(\"nrRB0.html\").find_all('p'):\n",
    "    print p.text\n",
    "    print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While this allows us to easily traverse the DOM and find specific elements by their id, class, or element type - we still have a lot of cruft in the document. This is where `readability-lxml` comes in. This library is a Python port of the [readability project](http://lab.arc90.com/experiments/readability/), written in Ruby and inspired by Instapaper. This code uses readability.js and some other helper functions to extract the main body and even title of the document you're working with. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A crisp and juicy bucket list of D.C.’s best fried chicken - The Washington Post\n"
     ]
    }
   ],
   "source": [
    "from readability.readability import Document\n",
    "\n",
    "def get_paper(path):\n",
    "    with codecs.open(path, 'r', encoding='utf-8') as f:\n",
    "        return Document(f.read())\n",
    "\n",
    "paper = get_paper(\"nrRB0.html\")\n",
    "print paper.title()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "with codecs.open(\"nrRB0-clean.html\", \"w\", encoding='utf-8') as f:\n",
    "    f.write(paper.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Combine readability and BeautifulSoup as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_text(path):\n",
    "    with open(path, 'r') as f:\n",
    "        paper = Document(f.read())\n",
    "        soup = bs4.BeautifulSoup(paper.summary())\n",
    "        output = [paper.title()]\n",
    "        for p in soup.find_all('p'):\n",
    "            output.append(p.text)\n",
    "        return \"\\n\\n\".join(output)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A crisp and juicy bucket list of D.C.’s best fried chicken - The Washington Post\n",
      "\n",
      "\n",
      "\n",
      " It’s lowbrow. It’s messy. It could never be accused of being healthful. But we’d never let those formalities get between us and an order of crispy, crackly, delicious fried chicken. Whether it comes in a bucket or on a bun, or you eat it with your fingers or chopsticks, there’s a surprising variety to the Washington area’s fried chicken offerings. Here are some of the most irresistible. \n",
      "\n",
      "\n",
      "\n",
      "‘Rotissi-fried’ chicken at the Partisan\n",
      "\n",
      "Forget the cronut. Our newest favorite food chimera is the “rotissi-fried” chicken at the Partisan. Credit goes to chef Nate Anda, who dreamed up the dish: After a 12-hour brine, the chicken is rotisseried for two hours and then fried for two and a half minutes. Why both? “Everything is better once it’s fried in beef fat,” Anda said. We have to agree. Whether white or dark, the meat is succulent throughout. The batter-free frying leaves the simply seasoned skin rendered perfectly crisp, golden and translucent — cracklings, essentially. The sound of it shattering under the knife was music to our ears. And as if the lily needed further gilding, the chicken comes with a generous pour of honey hot sauce. The sauce is hard to resist, but try to reserve a few bites of unadorned chicken so you can fully appreciate this happy marriage of classic preparations.\n",
      "\n",
      " The Partisan, 709 D St. NW. 202-524-5322. www.thepartisandc.com.  \n",
      "\n",
      "— Becky Krystal\n",
      "\n",
      "Traditional fried chicken at Family Meal\n",
      "\n",
      "When Bryan Voltaggio started planning the menu for Family Meal, his modern, upscale spin on a diner, his thoughts turned to home and the carryout meal he most enjoyed as a kid: fried chicken. “It was one of our favorite things,” he says. “It just seems like a family dinner.” And if he was creating a restaurant called Family Meal, fried chicken “had to be an important part of it.” But Voltaggio wanted to do it right, and set about testing cooking methods, brines, breadings and fryers. That long process paid off with a home run of a fried chicken dish that’s become the most popular item on Family Meal’s menu. The whole chickens spend 12 hours in a brine of pickle juice and roasted poultry stock before getting dredged, rested and dredged again in a mixture of flour, cornmeal and corn starch. After a dip in the top-of-the-line pressure fryer, the thighs, legs and breasts emerge with a crisp, salty skin that cracks open to reveal wonderfully warm and moist flesh. You don’t even need to dunk it in the house-made hot sauce that accompanies the dish, but really, who can resist? \n",
      "\n",
      " Family Meal Ashburn, 20462 Exchange St. 703-726-9800; Frederick, 880 N. East St. 301-378-2895; Baltimore, 621 E. Pratt St. 410-601-3242. www.voltfamilymeal.com.  \n",
      "\n",
      "— John Taylor\n",
      "\n",
      "Japanese fried chicken at Izakaya Seki\n",
      "\n",
      "Although it’s commonly served in Japan at karaoke bars, convenience stores and on street carts, kara age chicken — like most of the country’s food — is held to an extremely high standard. “It’s taken it to the nth degree of obsession and detail,” says Cizuka Seki, who, with her father Hiroshi, owns Izakaya Seki on V Street NW. “Kara age” is used to describe the method for deep-frying bite-size pieces of fish and, more commonly, chicken. Though there are subtle variations on the ubiquitous dish, most recipes call for chicken thighs marinated in soy sauce, coated in flour or corn starch and deep-fried in oil. Izakaya Seki’s version sticks closely to the formula. Probably. “I’m not even quite sure what my dad puts into it, because we don’t have recipes,” Seki says, though she’s certain wheat flour is involved. The result is a thin, tender coating that’s slightly softer than tempura. The accompanying ponzu sauce lends a tartness to the nubs.\n",
      "\n",
      " Izakaya Seki, 1117 V St. NW. 202-588-5841. www.sekidc.com.  \n",
      "\n",
      "— Holley Simmons\n",
      "\n",
      "Korean fried chicken at BonChon\n",
      "\n",
      "Don’t waste your kimchi-stinking breath asking for more sauce at BonChon. The South Korean fried chicken chain, founded in 2002, is so dedicated to consistency that it doesn’t allow for any modifications. And why would you want to change anything, really? The made-to-order wings, drumsticks and strips are fried twice, resulting in a paper-thin crust that yields the same satisfying crack as shattering crème brulee with a spoon. Founder Jinduk Seh spent two years perfecting his secret sauces, which come in three flavors — soy garlic, hot and a blend of the two — and are brushed on by hand post-fry, piece by piece. True to BonChon’s commitment to uniformity, sauces are made exclusively in South Korea and distributed to all 140-plus BonChon locations, which means the wings you’re chewing on in Arlington are slathered with the same exact stuff as those in the Philippines. Joints like these are so common throughout Korea they’re called “chimeks,” which is a hybrid term that combines “chicken” with the Korean word for beer. Washington should be happy to have 10 BonChons within driving distance, plus a brand new Metro-accessible location near the Navy Yard.\n",
      "\n",
      " BonChon, 1015 Half St. SE and nine other locations in Maryland and Virginia. www.bonchon.com.  \n",
      "\n",
      "— Holley Simmons\n",
      "\n",
      "Maryland fried chicken at Crisfield Seafood and Hank’s Oyster Bar\n",
      "\n",
      "There’s not much agreement on what constitutes Maryland fried chicken. Some say it’s just a fresh Maryland chicken that’s pan-fried; others say it should be topped with white gravy, almost like a chicken-fried steak. The pan-fried chicken platter at  Crisfield Seafood is a perfect example of the former style. Half of a chicken is dredged in flour, dusted with salt and pepper, and fried in a cast-iron pan. This preparation lends a snap and crunch to the exterior, and while the meat falls off the bone, the well-seasoned breading holds on. (The chicken is available only Friday through Sunday, and frequently sells out.) The Chesapeake fried chicken at Hank’s Oyster Bar in Dupont Circle and Capitol Hill is plumper than Crisfield’s version and seasoned with Old Bay, black pepper and cayenne, but the breading is softer and less crispy. It’s brined for 24 hours and deep-fried, rather than pan-fried, and it’s served only on Sunday. \n",
      "\n",
      " Crisfield Seafood, 8012 Georgia Ave., Silver Spring. 301-589-1306. www.crisfieldseafood.com. Hank’s Oyster Bar, 1624 Q St. NW. 202-462-4265; 633 Pennsylvania Ave. SE. 202-733-1971. www.hanksoysterbar.com. \n",
      "\n",
      "— Fritz Hahn\n",
      "\n",
      "Fancy fried chicken at Central\n",
      "\n",
      "Self-consciousness may prevent you from ordering fried chicken in a white-tablecloth restaurant. It feels incongruous — gauche, almost — to dig into picnic fare at the kind of place where you should be ordering risotto or tartare or something that comes with mousse, gelee or foam. But you have to override that adult voice in the back of your head, because if you don’t, you’ll miss out on Central Michel Richard’s famed fried chicken plate ($24 at lunch, $25 for dinner), which remains as good as ever. Though it’s no longer sold by the bucket to go, Michel Richard’s KFC-inspired crispy breast and thigh come stacked atop a pool of the butteriest mashed potatoes you’ll ever taste. It’s such a dignified presentation that this most American of dishes almost could pass as (gasp!) French. Self-consciousness should, however, steer you toward using a knife and fork — not your fingers — to eat the chicken. It is, after all, that kind of a place. \n",
      "\n",
      " Central Michel Richard, 1001 Pennsylvania Ave. NW. 202-626-0015. www.centralmichelrichard.com .   \n",
      "\n",
      "\n",
      "\n",
      "— Maura Judkis\n",
      "\n",
      "\n",
      "\n",
      "Nashville hot chicken at Reserve 2216\n",
      "\n",
      "If you believe the lore, Nashville hot chicken was basically a crime of passion, created as a blistering rebuke to a no-account Romeo who couldn’t keep his hands off other women. Alas, this wolf was also a chili head who found pleasure, not pain, in this dish of revenge served hot. Decades later, chefs are latching onto this addictive form of punishment. Aaron Silverman served an ultra-refined version at Rose’s Luxury for months, and now Eric Reid, chef and co-owner of Reserve 2216 in Del Ray, has developed his own take on hot chicken, even if he’s never actually enjoyed it in Nashville. He marinates an airline cut (boneless breast with the drumette wing attached) in buttermilk and Crystal hot sauce before dredging the chicken in seasoned flour and frying it. Reid ditches the traditional white bread base in favor of collards and a side of corn bread waffles. He finishes the dish with a combination of Cajun seasonings and more Crystal hot sauce for a moist, crispy bird that bites back. But not too hard. This is Alexandria, after all. \n",
      "\n",
      " Reserve 2216, 2216 Mount Vernon Ave., Alexandria. 703-549-2889. www.drpreserve.com. \n",
      "\n",
      "— Tim Carman\n",
      "\n",
      "\n",
      "\n",
      "Fast-food fried chicken at Popeyes\n",
      "\n",
      "The sole virtue of most fast-food operations is consistency. Whether you bite into a Big Mac in Bethesda or Beijing, the sandwich should taste the same. The menu at Popeyes follows suit, but it deviates from the brand-name competition in an important respect: The signature at Popeyes could pass for home cooking (well, if your home had a vat of clean, hot oil and a person with a Southern accent tending the meal). Maybe that accounts for my occasional forays to the chicken fryer after a bum restaurant-review excursion. No matter where I eat my order, inevitably “spicy,” I know I can count on a coating that smacks of cayenne, paprika and even crushed cornflakes, and chicken that spurts with juice. The shatter is audible; the golden crumbs fly everywhere, but end up on my tongue. No one-trick pony, Popeyes has hot and tender buttermilk biscuits that bolster my favorite excuse to snack low on the food chain. Once, I got home to discover a clerk had forgotten to pack bread in my bag. I almost cried. Instead, I consoled myself with another piece of chicken. \n",
      "\n",
      " Popeyes has locations throughout the D.C. metro area. www.popeyes.com. \n",
      "\n",
      "— Tom Sietsema\n",
      "\n",
      "\n",
      "\n",
      "Fried chicken sandwich at DCity Smokehouse\n",
      "\n",
      "The fried chicken sandwich hasn’t been the same since KFC’s Double Down turned a guilty pleasure into an outright farce, using crispy white-meat fillets as both the sandwich’s primary protein and the oily handles by which you eat the monstrosity. Leave it to Rob Sonderman, pitmaster and co-owner of DCity Smokehouse, to bring dignity back to the bite. His Den-Den — named for co-creator and pitmaster-in-training Dennis Geddie — begins with boneless thighs marinated in buttermilk, hot sauce and honey. Sonderman then dredges the meat in seasoned flour before dropping the thighs into the fryer. Generously stuffed into a grilled hoagie roll with lettuce, tomato and crispy onions, the chicken is finished with two sauces, including a house-made cilantro ranch. Technically, the Den-Den ($9.25) is one of the few smokeless items on DCity’s menu (unless you count the chipotle peppers in the hot sauce). No matter. You won’t care the minute you sink your teeth into that heaping hoagie of spicy thigh meat.\n",
      "\n",
      " DCity Smokehouse, 8 Florida Ave. NW. 202-733-1919. www.dcitysmokehouse.com.  \n",
      "\n",
      "— Tim Carman\n",
      "\n",
      "\n",
      "\n",
      "Classic D.C. fried chicken at Oohh’s and Aahh’s\n",
      "\n",
      "Hearty is the appetite that can handle Oohh’s and Aahh’s chef-owner Oji Abbott’s boneless fried chicken breast without taking home leftovers. He buys local — from Hartman Meat Co. in Northeast Washington — and butterflies each 14-ounce portion ($12.95), which results in a lot of real estate for the crispy, well-seasoned coating. Abbott chalks up the consistently moist meat to proper cooking time and temperature, and to the recipe he learned from his grandmother.\n",
      "\n",
      " Oohh’s and Aahh’s, 1005 U St. NW. 202-667-7142. www.oohhsnaahhs.com. \n",
      "\n",
      "— Bonnie S. Benwick\n",
      "\n",
      "\n",
      "\n",
      "Popcorn fried chicken at Pop’s Sea Bar\n",
      "\n",
      "It’s all too easy to chomp through an order of Boardwalk Chicken at the shore-happy Pop’s Sea Bar in Adams Morgan. The bite-size pieces of dark-meat-only bird are brined for two hours, then treated to a buttermilk bath until they are coated with plain flour and flash-fried to order. A generous hand with salt and pepper just before serving means the effect of that seasoning builds as you empty the single-serving basket ($8.99). Ask for two portions of the accompanying Jersey sauce so you’ll have enough of its horseradish-y, kitchen-sink blend for every bite.\n",
      "\n",
      " Pop’s Sea Bar, 1817 Columbia Rd. NW. 202-534-3933. www.popsseabar.com. \n",
      "\n",
      "— Bonnie S. Benwick\n",
      "\n",
      "\n",
      "\n",
      "Fried chicken tenders at GBD\n",
      "\n",
      "Chicken tenders are often relegated to the children’s menu, but the chicken Tendies at GBD are a fine meal for adults and children alike. Each white-meat tender comes with a dark, crispy outer layer with a heavy dose of salt and spice. But the best part about chicken tenders is the dipping, and GBD offers nine sauces, including a take on D.C.’s own mumbo sauce, buttermilk ranch, chipotle barbecue and Frankenbutter, which combines Frank’s RedHot sauce with butter. Ask for the $5.50 Saucetown option to try all nine. \n",
      "\n",
      " GBD, 1323 Connecticut Ave. NW. 202-524-5210. www.gbdchickendoughnuts.com. \n",
      "\n",
      "— Margaret Ely\n",
      "\n",
      "\n",
      "\n",
      "Fried chicken skins at Gypsy Soul\n",
      "\n",
      "Fried chicken fans can argue the merits of white meat vs. dark meat, or whether it’s better to chow down on a drumstick or the breast. But one thing we all can agree on is that the outer layer — the breading and the skin — is the most important element of a memorable piece of fried chicken. And sometimes you just want to savor the flavor of the skin — deeply spiced, perfect crunch — without filling up on meat or having to deal with bones. And that is when you grab one of the bar stools at R.J. Cooper’s Gypsy Soul in Fairfax’s Mosaic District. Cooper’s chicken skins ($9) are twisted slivers and shards, decadently salty and crackling with paprika, cayenne pepper and garlic. The dish arrives with a house-made “roof top honey-snake oil,” but it’s best to let these beauties shine on their own. \n",
      "\n",
      " Gypsy Soul, 8296 Glass Alley, Fairfax. 703-992-0933. www.gypsysoul-va.com. \n",
      "\n",
      "— Fritz Hahn\n"
     ]
    }
   ],
   "source": [
    "print get_text(\"nrRB0.html\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A note on binary formats\n",
    "\n",
    "In order to transform PDF documents to XML, the best solution is currently [PDFMiner](http://www.unixuser.org/~euske/python/pdfminer/index.html), specificially their [pdf2text](https://github.com/euske/pdfminer/blob/master/tools/pdf2txt.py) tool. Note that this tool can output into multiple formats like XML or HTML, which is often better than the direct text export. Because of this it's often useful to convert PDF to XHTML and then use Readabiilty or BeautifulSoup to extract the text out of the document. \n",
    "\n",
    "Unfortunately, the conversion from PDF to text is often not great, though statistical methodologies can help ease some of the errors in transformation. If PDFMiner is not sufficient, you can use tools like [PyPDF2](https://github.com/mstamy2/PyPDF2) to work directly on the PDF file, or write Python code to wrap other tools in Java and C like [PDFBox](https://pdfbox.apache.org/). \n",
    "\n",
    "Older binary formats like Pre-2007 Microsoft Word Documents (.doc) require special tools. Again, the best bet is to use Python to call another command line tool like [antiword](http://www.winfield.demon.nl/). Newer Microsoft formats are acutally zipped XML files (.docx) and can be either unzipped and handled using the XML tools mentioned above, or using Python packages like [python-docx](https://github.com/mikemaccana/python-docx) and [python-excel](http://www.python-excel.org/). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pattern\n",
    "\n",
    "The `pattern` library by the CLiPS lab at the University of Antwerp is designed specifically for language processing of web data and contains a toolkit for fetching data via web APIS: Google, Gmail, Bing, Twitter, Facebook, Wikipedia, and more. It supports HTML DOM parsing and even includes a web crawler! \n",
    "\n",
    "For example to ingest Twitter data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from pattern.web import Twitter, plaintext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RT @MicrosoftR: MT @DataEducationDC: Register for our #rstats special event w/ @RevoJoe 4/12 @WeWork in Dupont: https://t.co/2L75cA781v #da…\n",
      "RT @wahalulu: More #datadc at #rstatsnyc. @robertvesco @HarlanH https://t.co/1PfLS9w351\n",
      "RT @wahalulu: More #datadc at #rstatsnyc. @robertvesco @HarlanH https://t.co/1PfLS9w351\n",
      "More #datadc at #rstatsnyc. @robertvesco @HarlanH https://t.co/1PfLS9w351\n",
      "RT @wahalulu: Getting the #datadc gang together at #rstatsnyc  @robertvesco https://t.co/qebNLGWFFd\n",
      "Getting the #datadc gang together at #rstatsnyc  @robertvesco https://t.co/qebNLGWFFd\n",
      "RT @robertvesco: Screw teddy bears and dolls -&gt; Awesome plush statistical distributions via @NausicaaDist https://t.co/zHJiLQMfba #rstatsny…\n",
      "RT @tonyojeda3: Natural Language Processing with Python Workshop on 4/9 https://t.co/3gzxbXW2Jw #DataScience #BigData #NLProc #DataDC #DCTe…\n",
      "Screw teddy bears and dolls -&gt; Awesome plush statistical distributions via @NausicaaDist https://t.co/zHJiLQMfba #rstatsnyc #datadc\n",
      "Natural Language Processing with Python Workshop on 4/9 https://t.co/FcDwuReMoA #DataScience #BigData #NLProc #DataDC #DCTech #NLTK\n"
     ]
    }
   ],
   "source": [
    "twitter = Twitter(language='en')\n",
    "for tweet in twitter.search(\"#DataDC\", cached=False):\n",
    "    print tweet.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pattern also contains an NLP toolkit for English in the `pattern.en` module that utilizes statistical approcahes and regular expressions. Other languages include Spanish, French, Italian, German, and Dutch. \n",
    "\n",
    "The patern parser will identify word classes (e.g. Part of Speech tagging), perform morphological inflection analysis, and includes a WordNet API for lemmatization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The/DT/B-NP/O/NP-SBJ-1/the man/NN/I-NP/O/NP-SBJ-1/man hit/VBD/B-VP/O/VP-1/hit the/DT/O/O/O/the building/VBG/B-VP/O/O/build with/IN/B-PP/B-PNP/O/with a/DT/B-NP/I-PNP/O/a baseball/NN/I-NP/I-PNP/O/baseball bat/NN/I-NP/I-PNP/O/bat ././O/O/O/.\n",
      "\n",
      "Word(u'The/DT') Word(u'man/NN')\n",
      "Word(u'hit/VBD')\n",
      "Word(u'building/VBG')\n",
      "Word(u'with/IN')\n",
      "Word(u'a/DT') Word(u'baseball/NN') Word(u'bat/NN')\n"
     ]
    }
   ],
   "source": [
    "from pattern.en import parse, parsetree\n",
    "\n",
    "s = \"The man hit the building with a baseball bat.\"\n",
    "print parse(s, relations=True, lemmata=True)\n",
    "print\n",
    "for clause in parsetree(s):\n",
    "    for chunk in clause.chunks:\n",
    "        for word in chunk.words:\n",
    "            print word,\n",
    "        print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `pattern.search` module allows you to retreive N-Grams from text based on phrasal patterns, and can be used to mine dependencies from text, e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Match(words=[Word(u'The/DT'), Word(u'man/NN'), Word(u'hit/VBD')])\n"
     ]
    }
   ],
   "source": [
    "from pattern.search import search\n",
    "\n",
    "s = \"The man hit the building with a baseball bat.\"\n",
    "pt = parsetree(s, relations=True, lemmata=True)\n",
    "for match in search('NP VP', pt):\n",
    "    print match"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lastly the `pattern.vector` module has a toolkit for distance-based bag-of-words model machine learning including clustering (K-Means, Hierarhcical Clustering) and classification. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## NLTK\n",
    "\n",
    "Suite of libraries for a variety of academic text processing tasks:\n",
    "\n",
    "    tokenization, stemming, tagging,\n",
    "    chunking, parsing, classification,\n",
    "    language modeling, logical semantics\n",
    "\n",
    "Pedagogical resources for teaching NLP theory in Python ...\n",
    "\n",
    "- Python interface to over 50 corpora and lexical resources\n",
    "- Focus on Machine Learning with specific domain knowledge\n",
    "- Free and Open Source\n",
    "- Numpy and Scipy under the hood\n",
    "- Fast and Formal\n",
    "\n",
    "What is NLTK not?\n",
    "\n",
    "- Production ready out of the box*\n",
    "- Lightweight\n",
    "- Generally applicable\n",
    "- Magic\n",
    "\n",
    "*There are actually a few things that are production ready right out of the box*.\n",
    "\n",
    "**The Good Parts**:\n",
    "\n",
    "- Preprocessing\n",
    "    - segmentation\n",
    "    - tokenization\n",
    "    - PoS tagging\n",
    "- Word level processing\n",
    "    - WordNet\n",
    "    - Lemmatization\n",
    "    - Stemming\n",
    "    - NGram\n",
    "- Utilities\n",
    "    - Tree\n",
    "    - FreqDist\n",
    "    - ConditionalFreqDist\n",
    "- Streaming CorpusReader objects\n",
    "- Classification\n",
    "    - Maximum Entropy (Megam Algorithm)\n",
    "    - Naive Bayes\n",
    "    - Decision Tree\n",
    "- Chunking, Named Entity Recognition\n",
    "- Parsers Galore!\n",
    "\n",
    "**The Bad Parts**:\n",
    "\n",
    "- Syntactic Parsing\n",
    "    - No included grammar (not a black box)\n",
    "- Feature/Dependency Parsing\n",
    "    - No included feature grammar\n",
    "- The sem package\n",
    "- Toy only (lambda-calculus & first order logic)\n",
    "- Lots of extra stuff\n",
    "    - papers, chat programs, alignments, etc.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(u'A', 'DT'), (u'crisp', 'NN'), (u'and', 'CC'), (u'juicy', 'NN'), (u'bucket', 'NN'), (u'list', 'NN'), (u'of', 'IN'), (u'D', 'NNP'), (u'.', '.'), (u'C', 'NNP'), (u'.\\u2019', 'NNP'), (u's', 'VBZ'), (u'best', 'JJS'), (u'fried', 'VBN'), (u'chicken', 'NN'), (u'-', ':'), (u'The', 'DT'), (u'Washington', 'NNP'), (u'Post', 'NNP'), (u'It', 'NNP'), (u'\\u2019', 'NNP'), (u's', 'VBZ'), (u'lowbrow', 'NN'), (u'.', '.')]\n",
      "\n",
      "[(u'It', 'PRP'), (u'\\u2019', 'VBP'), (u's', 'NNS'), (u'messy', 'JJ'), (u'.', '.')]\n",
      "\n",
      "[(u'It', 'PRP'), (u'could', 'MD'), (u'never', 'RB'), (u'be', 'VB'), (u'accused', 'VBN'), (u'of', 'IN'), (u'being', 'VBG'), (u'healthful', 'JJ'), (u'.', '.')]\n",
      "\n",
      "[(u'But', 'CC'), (u'we', 'PRP'), (u'\\u2019', 'VBP'), (u'd', 'VBN'), (u'never', 'RB'), (u'let', 'VB'), (u'those', 'DT'), (u'formalities', 'NNS'), (u'get', 'VBP'), (u'between', 'IN'), (u'us', 'PRP'), (u'and', 'CC'), (u'an', 'DT'), (u'order', 'NN'), (u'of', 'IN'), (u'crispy', 'NN'), (u',', ','), (u'crackly', 'RB'), (u',', ','), (u'delicious', 'JJ'), (u'fried', 'JJ'), (u'chicken', 'NN'), (u'.', '.')]\n",
      "\n",
      "[(u'Whether', 'IN'), (u'it', 'PRP'), (u'comes', 'VBZ'), (u'in', 'IN'), (u'a', 'DT'), (u'bucket', 'NN'), (u'or', 'CC'), (u'on', 'IN'), (u'a', 'DT'), (u'bun', 'NN'), (u',', ','), (u'or', 'CC'), (u'you', 'PRP'), (u'eat', 'VBP'), (u'it', 'PRP'), (u'with', 'IN'), (u'your', 'PRP$'), (u'fingers', 'NNS'), (u'or', 'CC'), (u'chopsticks', 'NNS'), (u',', ','), (u'there', 'EX'), (u'\\u2019', ':'), (u's', 'NNS'), (u'a', 'DT'), (u'surprising', 'JJ'), (u'variety', 'NN'), (u'to', 'TO'), (u'the', 'DT'), (u'Washington', 'NNP'), (u'area', 'NN'), (u'\\u2019', ':'), (u's', 'NNS'), (u'fried', 'VBD'), (u'chicken', 'VBN'), (u'offerings', 'NNS'), (u'.', '.')]\n",
      "\n",
      "[(u'Here', 'RB'), (u'are', 'VBP'), (u'some', 'DT'), (u'of', 'IN'), (u'the', 'DT'), (u'most', 'RBS'), (u'irresistible', 'JJ'), (u'.', '.')]\n",
      "\n",
      "[(u'\\u2018', 'NN'), (u'Rotissi', 'NNP'), (u'-', ':'), (u'fried', 'VBD'), (u'\\u2019', 'CD'), (u'chicken', 'VBN'), (u'at', 'IN'), (u'the', 'DT'), (u'Partisan', 'NNP'), (u'Forget', 'NNP'), (u'the', 'DT'), (u'cronut', 'NN'), (u'.', '.')]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "text = get_text(\"nrRB0.html\")\n",
    "for idx, s in enumerate(nltk.sent_tokenize(text)): # Segmentation\n",
    "    words = nltk.wordpunct_tokenize(s)  # Tokenization\n",
    "    tags  = nltk.pos_tag(words)    # Part of Speech tagging\n",
    "    print tags\n",
    "    print\n",
    "    if idx > 5:\n",
    "        break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<FreqDist with 1072 samples and 3084 outcomes>\n",
      "<FreqDist with 1032 samples and 3084 outcomes>\n"
     ]
    }
   ],
   "source": [
    "from nltk import FreqDist\n",
    "from nltk import WordNetLemmatizer\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "text  = get_text(\"nrRB0.html\")\n",
    "vocab = FreqDist()\n",
    "words = FreqDist()\n",
    "for s in nltk.sent_tokenize(text): \n",
    "    for word in nltk.wordpunct_tokenize(s):\n",
    "        words[word] += 1\n",
    "        lemma = lemmatizer.lemmatize(word)\n",
    "        vocab[lemma] += 1\n",
    "\n",
    "print words\n",
    "print vocab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first thing you needed to do was create a corpus reader that could read the RSS feeds and their topics, implementing one of the built-in corpus readers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import nltk\n",
    "import time\n",
    "import random\n",
    "import pickle\n",
    "import string\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "from nltk.corpus import CategorizedPlaintextCorpusReader\n",
    "\n",
    "# The first group captures the category folder, docs are any HTML file.\n",
    "CORPUS_ROOT = './corpus'\n",
    "DOC_PATTERN = r'(?!\\.).*\\.html'\n",
    "CAT_PATTERN = r'([a-z_]+)/.*'\n",
    "\n",
    "# Specialized Corpus Reader for HTML documents\n",
    "class CategorizedHTMLCorpusreader(CategorizedPlaintextCorpusReader):\n",
    "    \"\"\"\n",
    "    Reads only the HTML body for the words and strips any tags.\n",
    "    \"\"\"\n",
    "\n",
    "    def _read_word_block(self, stream):\n",
    "        soup = BeautifulSoup(stream, 'lxml')\n",
    "        return self._word_tokenizer.tokenize(soup.get_text())\n",
    "\n",
    "    def _read_para_block(self, stream):\n",
    "        soup  = BeautifulSoup(stream, 'lxml')\n",
    "        paras = []\n",
    "        piter = soup.find_all('p') if soup.find('p') else self._para_block_reader(stream)\n",
    "\n",
    "        for para in piter:\n",
    "            paras.append([self._word_tokenizer.tokenize(sent)\n",
    "                          for sent in self._sent_tokenizer.tokenize(para)])\n",
    "\n",
    "        return paras\n",
    "\n",
    "# Create our corpus reader\n",
    "rss_corpus = CategorizedHTMLCorpusreader(CORPUS_ROOT, DOC_PATTERN,\n",
    "                    cat_pattern=CAT_PATTERN, encoding='utf-8')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just to make things easy, I've also included all of the imports at the top of this snippet in case you're just copying and pasting. This should give you a corpus that is easily readable with the following properties:\n",
    "\n",
    "> RSS Corpus contains 5506 files in 11 categories\n",
    "> Vocab: 69642 in 1920455 words for a lexical diversity of 27.576\n",
    "\n",
    "This snippet demonstrates a choice I made - to override the `_read_word_block` and the `_read_para_block` functions in the `CategorizedPlaintextCorpusReader`, but of course you could have created your own `HTMLCorpusReader` class that implemented the categorization features.\n",
    "\n",
    "The next thing to do is to figure out how you will generate your featuresets, I hope that you used unigrams, bigrams, TF-IDF and others. The simplest thing to do is simply a bag of words approach, however I have ensured that this bag of words does not contain punctuation or stopwords, has been normalized to all lowercase and has been lemmatized to reduce the number of word forms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create feature extractor methodology\n",
    "def normalize_words(document):\n",
    "    \"\"\"\n",
    "    Expects as input a list of words that make up a document. This will\n",
    "    yield only lowercase significant words (excluding stopwords and\n",
    "    punctuation) and will lemmatize all words to ensure that we have word\n",
    "    forms that are standardized.\n",
    "    \"\"\"\n",
    "    stopwords  = set(nltk.corpus.stopwords.words('english'))\n",
    "    lemmatizer = nltk.stem.wordnet.WordNetLemmatizer()\n",
    "    for token in document:\n",
    "        token = token.lower()\n",
    "        if token in string.punctuation: continue\n",
    "        if token in stopwords: continue\n",
    "        yield lemmatizer.lemmatize(token)\n",
    "\n",
    "def document_features(document):\n",
    "    words = nltk.FreqDist(normalize_words(document))\n",
    "    feats = {}\n",
    "    for word in words.keys():\n",
    "        feats['contains(%s)' % word] = True\n",
    "    return feats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should save a training, devtest and test as pickles to disk so that you can easily work on your classifier without having to worry about the overhead of randomization. I went ahead and saved the features to disk; but if you're developing features then you'll only save the word lists to disk. Here are the functions both for generation and for loading the data sets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Took 26.951 seconds to generate datasets\n"
     ]
    }
   ],
   "source": [
    "def timeit(func):\n",
    "    def wrapper(*args, **kwargs):\n",
    "        start  = time.time()\n",
    "        result = func(*args, **kwargs)\n",
    "        delta  = time.time() - start\n",
    "        return result, delta\n",
    "    return wrapper\n",
    "\n",
    "@timeit\n",
    "def generate_datasets(test_size=550, pickle_dir=\".\"):\n",
    "    \"\"\"\n",
    "    Creates three data sets; a test set and dev test set of 550 documents\n",
    "    then a training set with the rest of the documents in the corpus. It\n",
    "    will then write the data sets to disk at the pickle_dir.\n",
    "    \"\"\"\n",
    "    documents = [(document_features(rss_corpus.words(fileid)), category)\n",
    "                    for category in rss_corpus.categories()\n",
    "                    for fileid in rss_corpus.fileids(category)]\n",
    "\n",
    "    random.shuffle(documents)\n",
    "\n",
    "    datasets = {\n",
    "        'test':     documents[0:test_size],\n",
    "        'devtest':  documents[test_size:test_size*2],\n",
    "        'training': documents[test_size*2:],\n",
    "    }\n",
    "\n",
    "    for name, data in datasets.items():\n",
    "        with open(os.path.join(pickle_dir, name+\".pickle\"), 'wb') as out:\n",
    "            pickle.dump(data, out)\n",
    "\n",
    "def load_datasets(pickle_dir=\".\"):\n",
    "    \"\"\"\n",
    "    Loads the randomly shuffled data sets from their pickles on disk.\n",
    "    \"\"\"\n",
    "\n",
    "    def loader(name):\n",
    "        path = os.path.join(pickle_dir, name+\".pickle\")\n",
    "        with open(path, 'rb') as f:\n",
    "            data = pickle.load(f)\n",
    "\n",
    "        return name, data\n",
    "\n",
    "    return dict(loader(name) for name in ('test', 'devtest', 'training'))\n",
    "\n",
    "# Using a time it decorator you can see that this saves you quite a few seconds:\n",
    "\n",
    "_, delta = generate_datasets(pickle_dir='datasets')\n",
    "print \"Took %0.3f seconds to generate datasets\" % delta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Last up is the building of the classifier. I used a maximum entropy classifier with the lemmatized word level features. Also note that I used the MEGAM algorithm to significantly speed up my training time:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Found megam: /Users/benjamin/bin/megam]\n",
      "[Found megam: /Users/benjamin/bin/megam]\n",
      "trained in 133.205 seconds\n",
      "test accuracy 82.36%\n",
      "   3.917 contains(comment)==True and label is 'data_science'\n",
      "   3.599 contains(...)==True and label is 'gaming'\n",
      "   3.573 contains(data)==True and label is 'data_science'\n",
      "   3.248 contains(book)==True and label is 'books'\n",
      "   2.984 contains(wired)==True and label is 'tech'\n",
      "   2.970 label is 'business'\n",
      "   2.836 contains(»)==True and label is 'business'\n",
      "   2.667 contains(game)==True and label is 'gaming'\n",
      "   2.481 contains(entrepreneur)==True and label is 'business'\n",
      "  -2.418 label is 'essays'\n",
      "   2.342 contains(facebook)==True and label is 'tech'\n",
      "   2.259 contains(read)==True and label is 'tech'\n",
      "   2.255 contains(...)==True and label is 'cinema'\n",
      "   2.229 contains(adafruit)==True and label is 'do_it_yourself'\n",
      "   2.186 contains(recipe)==True and label is 'cooking'\n",
      "   2.166 contains(film)==True and label is 'cinema'\n",
      "  -2.101 contains(read)==True and label is 'business'\n",
      "   2.067 contains(business)==True and label is 'business'\n",
      "  -1.991 contains(’)==True and label is 'sports'\n",
      "  -1.957 label is 'do_it_yourself'\n",
      "   1.953 contains(nfl)==True and label is 'sports'\n",
      "   1.922 contains(sweet)==True and label is 'cooking'\n",
      "   1.913 contains(...))==True and label is 'design'\n",
      "   1.860 contains(steam)==True and label is 'gaming'\n",
      "   1.839 contains(dish)==True and label is 'cooking'\n",
      "   1.834 contains(mail)==True and label is 'books'\n",
      "   1.769 contains(cup)==True and label is 'sports'\n",
      "   1.766 contains(e)==True and label is 'books'\n",
      "   1.731 contains(appeared)==True and label is 'tech'\n",
      "   1.728 contains(.»)==True and label is 'books'\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Exception RuntimeError: 'generator ignored GeneratorExit' in <generator object find_file_iter at 0x1246070a0> ignored\n"
     ]
    }
   ],
   "source": [
    "@timeit\n",
    "def train_classifier(training, path='classifier.pickle'):\n",
    "    \"\"\"\n",
    "    Trains the classifier and saves it to disk.\n",
    "    \"\"\"\n",
    "    classifier = nltk.MaxentClassifier.train(training,\n",
    "                algorithm='megam', trace=2, gaussian_prior_sigma=1)\n",
    "\n",
    "    with open(path, 'wb') as out:\n",
    "        pickle.dump(classifier, out)\n",
    "\n",
    "    return classifier\n",
    "\n",
    "datasets = load_datasets(pickle_dir='datasets')\n",
    "classifier, delta = train_classifier(datasets['training'])\n",
    "print \"trained in %0.3f seconds\" % delta\n",
    "\n",
    "testacc    = nltk.classify.accuracy(classifier, datasets['test']) * 100\n",
    "print \"test accuracy %0.2f%%\" % testacc\n",
    "\n",
    "classifier.show_most_informative_features(30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "cooking: 1.0000\n",
      "essays: 0.0000\n",
      "books: 0.0000\n",
      "do_it_yourself: 0.0000\n",
      "gaming: 0.0000\n",
      "design: 0.0000\n",
      "tech: 0.0000\n",
      "cinema: 0.0000\n",
      "data_science: 0.0000\n",
      "sports: 0.0000\n",
      "business: 0.0000\n",
      "\n",
      "  Feature                                          cooking  essays   books do_it_y\n",
      "  --------------------------------------------------------------------------------\n",
      "  contains(recipe)==True (1)                         2.186\n",
      "  contains(dish)==True (1)                           1.839\n",
      "  contains(food)==True (1)                           1.073\n",
      "  contains(chef)==True (1)                           1.026\n",
      "  contains(classic)==True (1)                        0.980\n",
      "  contains(spicy)==True (1)                          0.881\n",
      "  contains(flavor)==True (1)                         0.853\n",
      "  contains(stuffed)==True (1)                        0.845\n",
      "  contains(served)==True (1)                         0.841\n",
      "  contains(bar)==True (1)                            0.823\n",
      "  contains(fresh)==True (1)                          0.787\n",
      "  contains(tomato)==True (1)                         0.783\n",
      "  contains(cooking)==True (1)                        0.729\n",
      "  contains(friday)==True (1)                         0.717\n",
      "  contains(delicious)==True (1)                      0.690\n",
      "  contains(sauce)==True (1)                          0.649\n",
      "  contains(oil)==True (1)                            0.609\n",
      "  contains(—)==True (1)                              0.583\n",
      "  contains(meal)==True (1)                           0.580\n",
      "  contains(topped)==True (1)                         0.577\n",
      "  contains(combination)==True (1)                    0.567\n",
      "  contains(spiced)==True (1)                         0.562\n",
      "  contains(grilled)==True (1)                        0.557\n",
      "  contains(without)==True (1)                        0.524\n",
      "  contains(pickle)==True (1)                         0.522\n",
      "  contains(made)==True (1)                           0.500\n",
      "  contains(meat)==True (1)                           0.500\n",
      "  contains(kitchen)==True (1)                        0.499\n",
      "  contains().)==True (1)                             0.477\n",
      "  contains(perfect)==True (1)                        0.476\n",
      "  contains(filling)==True (1)                        0.458\n",
      "  contains(pepper)==True (1)                         0.436\n",
      "  contains(time)==True (1)                          -0.429\n",
      "  contains(crisp)==True (1)                          0.429\n",
      "  contains(dinner)==True (1)                         0.424\n",
      "  contains(kimchi)==True (1)                         0.419\n",
      "  contains(home)==True (1)                           0.406\n",
      "  contains(seasoning)==True (1)                      0.387\n",
      "  contains(favorite)==True (1)                       0.373\n",
      "  contains(roasted)==True (1)                        0.369\n",
      "  contains(butter)==True (1)                         0.353\n",
      "  contains(restaurant)==True (1)                     0.349\n",
      "  contains(bite)==True (1)                           0.345\n",
      "  contains(year)==True (1)                          -0.344\n",
      "  contains(corn)==True (1)                           0.343\n",
      "  label is 'cooking' (1)                            -0.337\n",
      "  contains(almost)==True (1)                         0.329\n",
      "  contains(say)==True (1)                           -0.328\n",
      "  contains(fish)==True (1)                           0.326\n",
      "  contains(come)==True (1)                           0.298\n",
      "  contains(spring)==True (1)                         0.295\n",
      "  contains(secret)==True (1)                         0.294\n",
      "  contains(taken)==True (1)                          0.292\n",
      "  contains(potato)==True (1)                         0.292\n",
      "  contains(variation)==True (1)                      0.291\n",
      "  contains(hot)==True (1)                            0.289\n",
      "  contains(.”)==True (1)                             0.288\n",
      "  contains(onion)==True (1)                          0.286\n",
      "  contains(side)==True (1)                           0.286\n",
      "  contains(satisfying)==True (1)                     0.285\n",
      "  contains(change)==True (1)                        -0.283\n",
      "  contains(pack)==True (1)                           0.281\n",
      "  contains(’)==True (1)                             -0.281\n",
      "  contains(salt)==True (1)                           0.281\n",
      "  contains(garlic)==True (1)                         0.280\n",
      "  contains(chicken)==True (1)                        0.279\n",
      "  contains(seafood)==True (1)                        0.277\n",
      "  contains(cayenne)==True (1)                        0.274\n",
      "  contains(country)==True (1)                        0.270\n",
      "  contains(version)==True (1)                        0.269\n",
      "  contains(mixture)==True (1)                        0.267\n",
      "  contains(juice)==True (1)                          0.267\n",
      "  contains(used)==True (1)                           0.260\n",
      "  contains(often)==True (1)                          0.259\n",
      "  contains(traditional)==True (1)                    0.250\n",
      "  contains(temperature)==True (1)                    0.249\n",
      "  contains(house)==True (1)                          0.247\n",
      "  contains(taste)==True (1)                          0.244\n",
      "  contains(snack)==True (1)                          0.241\n",
      "  contains(plate)==True (1)                          0.236\n",
      "  contains(good)==True (1)                           0.233\n",
      "  contains(fried)==True (1)                          0.227\n",
      "  contains(beauty)==True (1)                         0.226\n",
      "  contains(lettuce)==True (1)                        0.226\n",
      "  contains(lunch)==True (1)                          0.224\n",
      "  contains(right)==True (1)                          0.224\n",
      "  contains(flour)==True (1)                          0.223\n",
      "  contains(style)==True (1)                          0.222\n",
      "  contains(kind)==True (1)                           0.221\n",
      "  contains(‘)==True (1)                             -0.221\n",
      "  contains(risotto)==True (1)                        0.215\n",
      "  contains(roll)==True (1)                           0.214\n",
      "  contains(probably)==True (1)                       0.213\n",
      "  contains(open)==True (1)                          -0.211\n",
      "  contains(white)==True (1)                          0.210\n",
      "  contains(5)==True (1)                             -0.208\n",
      "  contains(variety)==True (1)                        0.208\n",
      "  contains(new)==True (1)                           -0.207\n",
      "  contains(want)==True (1)                          -0.202\n",
      "  contains(com)==True (1)                           -0.202\n",
      "  contains(warm)==True (1)                           0.200\n",
      "  contains(piece)==True (1)                         -0.199\n",
      "  contains(slathered)==True (1)                      0.199\n",
      "  contains(crispy)==True (1)                         0.199\n",
      "  contains(best)==True (1)                           0.197\n",
      "  contains(street)==True (1)                         0.195\n",
      "  contains(menu)==True (1)                           0.194\n",
      "  contains(much)==True (1)                           0.192\n",
      "  contains(head)==True (1)                           0.192\n",
      "  contains(e)==True (1)                             -0.190\n",
      "  contains(.))==True (1)                             0.184\n",
      "  contains(need)==True (1)                          -0.184\n",
      "  contains(8)==True (1)                             -0.183\n",
      "  contains(leave)==True (1)                          0.179\n",
      "  contains(le)==True (1)                            -0.178\n",
      "  contains(hour)==True (1)                          -0.176\n",
      "  contains(could)==True (1)                         -0.176\n",
      "  contains(child)==True (1)                         -0.174\n",
      "  contains(woman)==True (1)                         -0.174\n",
      "  contains(”)==True (1)                             -0.174\n",
      "  contains(steak)==True (1)                          0.174\n",
      "  contains(list)==True (1)                           0.171\n",
      "  contains(slightly)==True (1)                       0.170\n",
      "  contains(put)==True (1)                            0.169\n",
      "  contains(option)==True (1)                        -0.166\n",
      "  contains(former)==True (1)                        -0.163\n",
      "  contains(may)==True (1)                            0.163\n",
      "  contains(know)==True (1)                          -0.162\n",
      "  contains(dark)==True (1)                           0.160\n",
      "  contains(name)==True (1)                          -0.159\n",
      "  contains(hand)==True (1)                           0.158\n",
      "  contains(created)==True (1)                       -0.157\n",
      "  contains(throughout)==True (1)                     0.157\n",
      "  contains(month)==True (1)                         -0.155\n",
      "  contains(one)==True (1)                            0.153\n",
      "  contains(miss)==True (1)                          -0.152\n",
      "  contains(including)==True (1)                      0.151\n",
      "  contains(batter)==True (1)                         0.151\n",
      "  contains(shore)==True (1)                          0.151\n",
      "  contains(hard)==True (1)                          -0.150\n",
      "  contains(credit)==True (1)                         0.150\n",
      "  contains(really)==True (1)                         0.150\n",
      "  contains(area)==True (1)                          -0.149\n",
      "  contains(anything)==True (1)                       0.149\n",
      "  contains(another)==True (1)                       -0.149\n",
      "  contains(suit)==True (1)                           0.148\n",
      "  contains(pas)==True (1)                            0.147\n",
      "  contains(leaf)==True (1)                           0.146\n",
      "  contains(tender)==True (1)                         0.145\n",
      "  contains(honey)==True (1)                          0.145\n",
      "  contains(crust)==True (1)                          0.145\n",
      "  contains(oyster)==True (1)                         0.142\n",
      "  contains(part)==True (1)                           0.142\n",
      "  contains(begin)==True (1)                         -0.142\n",
      "  contains(v)==True (1)                             -0.142\n",
      "  contains(u)==True (1)                              0.142\n",
      "  contains(quite)==True (1)                          0.141\n",
      "  contains(deep)==True (1)                           0.138\n",
      "  contains(discover)==True (1)                      -0.137\n",
      "  contains(beer)==True (1)                           0.137\n",
      "  contains(,”)==True (1)                             0.135\n",
      "  contains(run)==True (1)                           -0.134\n",
      "  contains(grab)==True (1)                           0.132\n",
      "  contains(spice)==True (1)                          0.131\n",
      "  contains(sea)==True (1)                            0.131\n",
      "  contains(family)==True (1)                         0.130\n",
      "  contains(review)==True (1)                        -0.129\n",
      "  contains(blend)==True (1)                          0.129\n",
      "  contains(easy)==True (1)                           0.129\n",
      "  contains(whether)==True (1)                       -0.128\n",
      "  contains(modern)==True (1)                        -0.128\n",
      "  contains(co)==True (1)                            -0.128\n",
      "  contains(others)==True (1)                        -0.128\n",
      "  contains(standard)==True (1)                       0.126\n",
      "  contains(taking)==True (1)                        -0.125\n",
      "  contains(like)==True (1)                           0.125\n",
      "  contains(spend)==True (1)                         -0.124\n",
      "  contains(free)==True (1)                           0.123\n",
      "  contains(detail)==True (1)                         0.122\n",
      "  contains(dose)==True (1)                           0.122\n",
      "  contains(wanted)==True (1)                         0.121\n",
      "  contains(pan)==True (1)                            0.119\n",
      "  contains(thin)==True (1)                           0.119\n",
      "  contains(maybe)==True (1)                         -0.118\n",
      "  contains(person)==True (1)                        -0.118\n",
      "  contains(spent)==True (1)                          0.118\n",
      "  contains(longer)==True (1)                        -0.118\n",
      "  contains(line)==True (1)                           0.117\n",
      "  contains(french)==True (1)                        -0.117\n",
      "  contains(better)==True (1)                        -0.117\n",
      "  contains(self)==True (1)                          -0.116\n",
      "  contains(memorable)==True (1)                      0.116\n",
      "  contains(clean)==True (1)                         -0.114\n",
      "  contains(true)==True (1)                          -0.113\n",
      "  contains(founder)==True (1)                       -0.113\n",
      "  contains(started)==True (1)                        0.113\n",
      "  contains(frequently)==True (1)                     0.112\n",
      "  contains(keep)==True (1)                          -0.112\n",
      "  contains(age)==True (1)                           -0.111\n",
      "  contains(picnic)==True (1)                         0.111\n",
      "  contains(fat)==True (1)                            0.110\n",
      "  contains(fall)==True (1)                           0.109\n",
      "  contains(happy)==True (1)                         -0.108\n",
      "  contains(simply)==True (1)                         0.108\n",
      "  contains(leftover)==True (1)                       0.107\n",
      "  contains(getting)==True (1)                       -0.106\n",
      "  contains(),)==True (1)                             0.106\n",
      "  contains(fry)==True (1)                            0.104\n",
      "  contains(offer)==True (1)                          0.104\n",
      "  contains(j)==True (1)                             -0.102\n",
      "  contains(diner)==True (1)                          0.102\n",
      "  contains(competition)==True (1)                    0.102\n",
      "  contains(plus)==True (1)                          -0.101\n",
      "  contains(held)==True (1)                           0.100\n",
      "  contains(important)==True (1)                     -0.100\n",
      "  contains(spoon)==True (1)                          0.099\n",
      "  contains(serving)==True (1)                        0.099\n",
      "  contains(set)==True (1)                           -0.098\n",
      "  contains(rose)==True (1)                           0.098\n",
      "  contains(basically)==True (1)                      0.098\n",
      "  contains(try)==True (1)                            0.098\n",
      "  contains(bag)==True (1)                            0.097\n",
      "  contains(although)==True (1)                      -0.096\n",
      "  contains(degree)==True (1)                         0.095\n",
      "  contains(half)==True (1)                          -0.094\n",
      "  contains(thing)==True (1)                         -0.093\n",
      "  contains(american)==True (1)                      -0.093\n",
      "  contains(waste)==True (1)                          0.092\n",
      "  contains(seasoned)==True (1)                       0.091\n",
      "  contains(eat)==True (1)                            0.090\n",
      "  contains(place)==True (1)                         -0.090\n",
      "  contains(pain)==True (1)                           0.090\n",
      "  contains(sandwich)==True (1)                       0.089\n",
      "  contains(well)==True (1)                           0.089\n",
      "  contains(mousse)==True (1)                         0.088\n",
      "  contains(feel)==True (1)                           0.086\n",
      "  contains(strip)==True (1)                          0.086\n",
      "  contains(process)==True (1)                       -0.086\n",
      "  contains(real)==True (1)                          -0.086\n",
      "  contains(washington)==True (1)                     0.085\n",
      "  contains(combine)==True (1)                        0.085\n",
      "  contains(within)==True (1)                        -0.085\n",
      "  contains(never)==True (1)                         -0.085\n",
      "  contains(24)==True (1)                            -0.084\n",
      "  contains(term)==True (1)                          -0.084\n",
      "  contains(rendered)==True (1)                       0.083\n",
      "  contains(also)==True (1)                           0.083\n",
      "  contains(buy)==True (1)                            0.082\n",
      "  contains(whole)==True (1)                          0.081\n",
      "  contains(build)==True (1)                         -0.081\n",
      "  contains(result)==True (1)                        -0.080\n",
      "  contains(using)==True (1)                          0.080\n",
      "  contains(appreciate)==True (1)                     0.080\n",
      "  contains(thought)==True (1)                        0.080\n",
      "  contains(store)==True (1)                         -0.079\n",
      "  contains(50)==True (1)                            -0.077\n",
      "  contains(base)==True (1)                           0.077\n",
      "  contains(become)==True (1)                        -0.076\n",
      "  contains(john)==True (1)                          -0.076\n",
      "  contains(sometimes)==True (1)                      0.076\n",
      "  contains(layer)==True (1)                          0.075\n",
      "  contains(japanese)==True (1)                      -0.074\n",
      "  contains(beef)==True (1)                           0.073\n",
      "  contains(snap)==True (1)                           0.073\n",
      "  contains(would)==True (1)                          0.073\n",
      "  contains(dipping)==True (1)                        0.072\n",
      "  contains(trick)==True (1)                         -0.072\n",
      "  contains(care)==True (1)                          -0.072\n",
      "  contains(everything)==True (1)                     0.071\n",
      "  contains(crumb)==True (1)                          0.071\n",
      "  contains(single)==True (1)                        -0.071\n",
      "  contains(element)==True (1)                       -0.071\n",
      "  contains(near)==True (1)                          -0.070\n",
      "  contains(2002)==True (1)                           0.070\n",
      "  contains(gravy)==True (1)                          0.070\n",
      "  contains(found)==True (1)                          0.070\n",
      "  contains(music)==True (1)                         -0.070\n",
      "  contains(exclusively)==True (1)                    0.070\n",
      "  contains(wonderfully)==True (1)                    0.070\n",
      "  contains(hearty)==True (1)                         0.069\n",
      "  contains(12)==True (1)                            -0.068\n",
      "  contains(decade)==True (1)                        -0.068\n",
      "  contains(ask)==True (1)                           -0.068\n",
      "  contains(dusted)==True (1)                         0.068\n",
      "  contains(driving)==True (1)                       -0.067\n",
      "  contains(passion)==True (1)                        0.066\n",
      "  contains(!))==True (1)                             0.065\n",
      "  contains(order)==True (1)                          0.064\n",
      "  contains(refined)==True (1)                        0.063\n",
      "  contains(luxury)==True (1)                        -0.063\n",
      "  contains(example)==True (1)                       -0.062\n",
      "  contains(take)==True (1)                           0.062\n",
      "  contains(however)==True (1)                       -0.061\n",
      "  contains(item)==True (1)                          -0.061\n",
      "  contains(prevent)==True (1)                       -0.061\n",
      "  contains(stock)==True (1)                          0.059\n",
      "  contains(kid)==True (1)                           -0.059\n",
      "  contains(bring)==True (1)                         -0.059\n",
      "  contains(25)==True (1)                            -0.059\n",
      "  contains(black)==True (1)                          0.058\n",
      "  contains(wing)==True (1)                           0.058\n",
      "  contains(basket)==True (1)                         0.058\n",
      "  contains(cilantro)==True (1)                       0.057\n",
      "  contains(tom)==True (1)                            0.057\n",
      "  contains(mashed)==True (1)                         0.056\n",
      "  contains(fork)==True (1)                           0.056\n",
      "  contains(later)==True (1)                         -0.056\n",
      "  contains(fan)==True (1)                           -0.056\n",
      "  contains(lends)==True (1)                          0.055\n",
      "  contains(iron)==True (1)                          -0.054\n",
      "  contains(alike)==True (1)                          0.054\n",
      "  contains(fancy)==True (1)                          0.054\n",
      "  contains(soul)==True (1)                          -0.054\n",
      "  contains(signature)==True (1)                      0.053\n",
      "  contains(needed)==True (1)                        -0.053\n",
      "  contains(training)==True (1)                      -0.053\n",
      "  contains(location)==True (1)                      -0.052\n",
      "  contains(attached)==True (1)                      -0.052\n",
      "  contains(moist)==True (1)                          0.051\n",
      "  contains(sold)==True (1)                          -0.051\n",
      "  contains(frank)==True (1)                         -0.051\n",
      "  contains(actually)==True (1)                      -0.050\n",
      "  contains(minute)==True (1)                        -0.050\n",
      "  contains(subtle)==True (1)                        -0.050\n",
      "  contains(pleasure)==True (1)                      -0.049\n",
      "  contains(plain)==True (1)                          0.049\n",
      "  contains(cut)==True (1)                           -0.049\n",
      "  contains(believe)==True (1)                        0.049\n",
      "  contains(mean)==True (1)                           0.049\n",
      "  contains(enough)==True (1)                         0.048\n",
      "  contains(bay)==True (1)                           -0.047\n",
      "  contains(excursion)==True (1)                      0.047\n",
      "  contains(pool)==True (1)                          -0.046\n",
      "  contains(heavy)==True (1)                          0.046\n",
      "  contains(brand)==True (1)                          0.045\n",
      "  contains(c)==True (1)                              0.045\n",
      "  contains(south)==True (1)                          0.045\n",
      "  contains(offering)==True (1)                      -0.045\n",
      "  contains(crunch)==True (1)                         0.044\n",
      "  contains(generous)==True (1)                       0.044\n",
      "  contains(post)==True (1)                          -0.044\n",
      "  contains(seems)==True (1)                          0.044\n",
      "  contains(sound)==True (1)                         -0.044\n",
      "  contains(dip)==True (1)                            0.043\n",
      "  contains(st)==True (1)                             0.043\n",
      "  contains(fine)==True (1)                          -0.043\n",
      "  contains(sell)==True (1)                           0.043\n",
      "  contains(lot)==True (1)                            0.042\n",
      "  contains(owner)==True (1)                          0.042\n",
      "  contains(paid)==True (1)                           0.042\n",
      "  contains(nine)==True (1)                           0.042\n",
      "  contains(juicy)==True (1)                          0.042\n",
      "  contains(matter)==True (1)                         0.041\n",
      "  contains(technically)==True (1)                   -0.041\n",
      "  contains(kara)==True (1)                           0.041\n",
      "  contains(chili)==True (1)                         -0.040\n",
      "  contains(finished)==True (1)                       0.040\n",
      "  contains(toward)==True (1)                        -0.040\n",
      "  contains(georgia)==True (1)                        0.040\n",
      "  contains(teeth)==True (1)                          0.040\n",
      "  contains(form)==True (1)                           0.040\n",
      "  contains(old)==True (1)                           -0.040\n",
      "  contains(newest)==True (1)                        -0.039\n",
      "  contains(twice)==True (1)                          0.039\n",
      "  contains(exterior)==True (1)                      -0.039\n",
      "  contains(dedicated)==True (1)                      0.039\n",
      "  contains(even)==True (1)                           0.039\n",
      "  contains(father)==True (1)                        -0.039\n",
      "  contains(involved)==True (1)                      -0.038\n",
      "  contains(district)==True (1)                      -0.038\n",
      "  contains(adult)==True (1)                         -0.037\n",
      "  contains(count)==True (1)                         -0.037\n",
      "  contains(resist)==True (1)                        -0.037\n",
      "  contains(pour)==True (1)                          -0.037\n",
      "  contains(sure)==True (1)                           0.036\n",
      "  contains(available)==True (1)                     -0.036\n",
      "  contains(bone)==True (1)                          -0.036\n",
      "  contains(sunday)==True (1)                        -0.036\n",
      "  contains(joint)==True (1)                         -0.036\n",
      "  contains(surprising)==True (1)                    -0.035\n",
      "  contains(frying)==True (1)                        -0.035\n",
      "  contains(planning)==True (1)                      -0.034\n",
      "  contains(long)==True (1)                           0.034\n",
      "  contains(popular)==True (1)                        0.033\n",
      "  contains(agree)==True (1)                         -0.033\n",
      "  contains(taylor)==True (1)                         0.033\n",
      "  contains(florida)==True (1)                       -0.032\n",
      "  contains(testing)==True (1)                       -0.032\n",
      "  contains(stacked)==True (1)                       -0.032\n",
      "  contains(hold)==True (1)                          -0.032\n",
      "  contains(deal)==True (1)                          -0.032\n",
      "  contains(10)==True (1)                             0.032\n",
      "  contains(remains)==True (1)                        0.031\n",
      "  contains(golden)==True (1)                        -0.031\n",
      "  contains(obsession)==True (1)                     -0.031\n",
      "  contains(follows)==True (1)                       -0.031\n",
      "  contains(wheat)==True (1)                         -0.031\n",
      "  contains(presentation)==True (1)                  -0.030\n",
      "  contains(exact)==True (1)                          0.030\n",
      "  contains(.’)==True (1)                             0.030\n",
      "  contains(marriage)==True (1)                      -0.030\n",
      "  contains(let)==True (1)                           -0.029\n",
      "  contains(crack)==True (1)                         -0.029\n",
      "  contains(go)==True (1)                            -0.029\n",
      "  contains(“)==True (1)                              0.029\n",
      "  contains(back)==True (1)                          -0.029\n",
      "  contains(salty)==True (1)                          0.028\n",
      "  contains(ever)==True (1)                          -0.028\n",
      "  contains(central)==True (1)                       -0.028\n",
      "  contains(fully)==True (1)                         -0.027\n",
      "  contains(protein)==True (1)                        0.027\n",
      "  contains(creator)==True (1)                       -0.027\n",
      "  contains(perfectly)==True (1)                     -0.026\n",
      "  contains(q)==True (1)                              0.026\n",
      "  contains(relegated)==True (1)                      0.026\n",
      "  contains(developed)==True (1)                      0.026\n",
      "  contains(fillet)==True (1)                         0.025\n",
      "  contains(nashville)==True (1)                      0.025\n",
      "  contains(southern)==True (1)                       0.024\n",
      "  contains(yard)==True (1)                          -0.024\n",
      "  contains(proper)==True (1)                        -0.024\n",
      "  contains(effect)==True (1)                        -0.024\n",
      "  contains(finger)==True (1)                        -0.023\n",
      "  contains(brushed)==True (1)                       -0.023\n",
      "  contains(word)==True (1)                           0.022\n",
      "  contains(hybrid)==True (1)                        -0.022\n",
      "  contains(high)==True (1)                          -0.022\n",
      "  contains(stuff)==True (1)                         -0.021\n",
      "  contains(circle)==True (1)                        -0.021\n",
      "  contains(($)==True (1)                            -0.021\n",
      "  contains(two)==True (1)                           -0.021\n",
      "  contains(inspired)==True (1)                       0.021\n",
      "  contains(bucket)==True (1)                        -0.021\n",
      "  contains(outer)==True (1)                         -0.021\n",
      "  contains(rather)==True (1)                        -0.021\n",
      "  contains(unless)==True (1)                         0.020\n",
      "  contains(essentially)==True (1)                    0.020\n",
      "  contains(forget)==True (1)                        -0.020\n",
      "  contains(local)==True (1)                          0.020\n",
      "  contains(spin)==True (1)                          -0.020\n",
      "  contains(resulting)==True (1)                      0.020\n",
      "  contains(ordering)==True (1)                      -0.019\n",
      "  contains(skin)==True (1)                          -0.019\n",
      "  contains(pop)==True (1)                            0.019\n",
      "  contains(chain)==True (1)                          0.019\n",
      "  contains(allow)==True (1)                          0.019\n",
      "  contains(knife)==True (1)                          0.019\n",
      "  contains(everywhere)==True (1)                    -0.018\n",
      "  contains(roof)==True (1)                          -0.018\n",
      "  contains(big)==True (1)                           -0.018\n",
      "  contains(onto)==True (1)                          -0.018\n",
      "  contains(addictive)==True (1)                      0.018\n",
      "  contains(describe)==True (1)                      -0.018\n",
      "  contains(end)==True (1)                           -0.017\n",
      "  contains(three)==True (1)                         -0.017\n",
      "  contains(silver)==True (1)                         0.017\n",
      "  contains(paper)==True (1)                         -0.017\n",
      "  contains(excuse)==True (1)                        -0.017\n",
      "  contains(mount)==True (1)                         -0.017\n",
      "  contains(jersey)==True (1)                        -0.017\n",
      "  contains(virginia)==True (1)                       0.017\n",
      "  contains(generously)==True (1)                    -0.017\n",
      "  contains(exchange)==True (1)                      -0.016\n",
      "  contains(dunk)==True (1)                          -0.016\n",
      "  contains(ear)==True (1)                           -0.016\n",
      "  contains(.,)==True (1)                             0.016\n",
      "  contains(creating)==True (1)                       0.016\n",
      "  contains(northeast)==True (1)                      0.016\n",
      "  contains(cart)==True (1)                           0.015\n",
      "  contains(bird)==True (1)                          -0.015\n",
      "  contains(extremely)==True (1)                     -0.015\n",
      "  contains(14)==True (1)                            -0.015\n",
      "  contains(reserve)==True (1)                       -0.015\n",
      "  contains(named)==True (1)                         -0.014\n",
      "  contains(shine)==True (1)                         -0.014\n",
      "  contains(cast)==True (1)                          -0.014\n",
      "  contains(reveal)==True (1)                        -0.014\n",
      "  contains(asking)==True (1)                        -0.014\n",
      "  contains(agreement)==True (1)                     -0.014\n",
      "  contains(barbecue)==True (1)                       0.014\n",
      "  contains(favor)==True (1)                         -0.013\n",
      "  contains(glass)==True (1)                         -0.013\n",
      "  contains(breath)==True (1)                        -0.013\n",
      "  contains(9)==True (1)                              0.012\n",
      "  contains(though)==True (1)                        -0.012\n",
      "  contains(yield)==True (1)                          0.012\n",
      "  contains(cajun)==True (1)                          0.012\n",
      "  contains(commitment)==True (1)                    -0.012\n",
      "  contains(since)==True (1)                          0.012\n",
      "  contains(something)==True (1)                     -0.011\n",
      "  contains(accessible)==True (1)                     0.011\n",
      "  contains(sink)==True (1)                          -0.011\n",
      "  contains(ultra)==True (1)                         -0.011\n",
      "  contains(enjoyed)==True (1)                       -0.011\n",
      "  contains(top)==True (1)                           -0.011\n",
      "  contains(argue)==True (1)                          0.010\n",
      "  contains(tartness)==True (1)                      -0.010\n",
      "  contains(closely)==True (1)                       -0.010\n",
      "  contains(brine)==True (1)                          0.010\n",
      "  contains(dig)==True (1)                           -0.009\n",
      "  contains(korean)==True (1)                        -0.009\n",
      "  contains(low)==True (1)                           -0.009\n",
      "  contains(instead)==True (1)                        0.009\n",
      "  contains(get)==True (1)                           -0.009\n",
      "  contains(portion)==True (1)                       -0.009\n",
      "  contains(99)==True (1)                             0.009\n",
      "  contains(double)==True (1)                        -0.009\n",
      "  contains(margaret)==True (1)                       0.009\n",
      "  contains(ala)==True (1)                           -0.008\n",
      "  contains(consistency)==True (1)                    0.008\n",
      "  contains(mosaic)==True (1)                        -0.008\n",
      "  contains(owns)==True (1)                          -0.008\n",
      "  contains(size)==True (1)                           0.007\n",
      "  contains(occasional)==True (1)                    -0.007\n",
      "  contains(said)==True (1)                           0.007\n",
      "  contains(appetite)==True (1)                      -0.007\n",
      "  contains(healthful)==True (1)                      0.007\n",
      "  contains(bath)==True (1)                          -0.007\n",
      "  contains(alley)==True (1)                         -0.006\n",
      "  contains(coated)==True (1)                         0.006\n",
      "  contains(tending)==True (1)                        0.006\n",
      "  contains(tongue)==True (1)                        -0.006\n",
      "  contains(dad)==True (1)                           -0.006\n",
      "  contains(learned)==True (1)                       -0.006\n",
      "  contains(famed)==True (1)                         -0.006\n",
      "  contains(fast)==True (1)                           0.005\n",
      "  contains(messy)==True (1)                         -0.005\n",
      "  contains(atop)==True (1)                          -0.005\n",
      "  contains(common)==True (1)                         0.005\n",
      "  contains(outright)==True (1)                       0.005\n",
      "  contains(distributed)==True (1)                   -0.005\n",
      "  contains(ave)==True (1)                            0.005\n",
      "  contains(finish)==True (1)                        -0.004\n",
      "  contains(formula)==True (1)                        0.004\n",
      "  contains(method)==True (1)                         0.004\n",
      "  contains(foam)==True (1)                          -0.004\n",
      "  contains(thigh)==True (1)                         -0.004\n",
      "  contains(mac)==True (1)                           -0.004\n",
      "  contains(biscuit)==True (1)                        0.004\n",
      "  contains(ounce)==True (1)                          0.004\n",
      "  contains(stick)==True (1)                         -0.003\n",
      "  contains(guilty)==True (1)                        -0.003\n",
      "  contains(call)==True (1)                           0.003\n",
      "  contains(bread)==True (1)                          0.003\n",
      "  contains(crushed)==True (1)                        0.003\n",
      "  contains(del)==True (1)                           -0.003\n",
      "  contains(commonly)==True (1)                       0.003\n",
      "  contains(east)==True (1)                          -0.003\n",
      "  contains(pressure)==True (1)                      -0.003\n",
      "  contains(got)==True (1)                            0.003\n",
      "  contains(accent)==True (1)                         0.003\n",
      "  contains(ubiquitous)==True (1)                     0.002\n",
      "  contains(upscale)==True (1)                       -0.002\n",
      "  contains(deeply)==True (1)                        -0.002\n",
      "  contains(n)==True (1)                              0.002\n",
      "  contains(softer)==True (1)                         0.002\n",
      "  contains(handle)==True (1)                         0.002\n",
      "  contains(preparation)==True (1)                    0.002\n",
      "  contains(sliver)==True (1)                        -0.002\n",
      "  contains(turned)==True (1)                        -0.002\n",
      "  contains(every)==True (1)                          0.002\n",
      "  contains(brined)==True (1)                         0.002\n",
      "  contains(savor)==True (1)                          0.001\n",
      "  contains(popcorn)==True (1)                       -0.001\n",
      "  contains(convenience)==True (1)                   -0.001\n",
      "  contains(fare)==True (1)                          -0.001\n",
      "  contains(tartare)==True (1)                        0.001\n",
      "  contains(bum)==True (1)                           -0.001\n",
      "  contains(flesh)==True (1)                         -0.001\n",
      "  contains(se)==True (1)                            -0.001\n",
      "  contains(perfecting)==True (1)                     0.001\n",
      "  contains(cornmeal)==True (1)                       0.001\n",
      "  contains(irresistible)==True (1)                   0.001\n",
      "  contains(called)==True (1)                         0.001\n",
      "  contains(tablecloth)==True (1)                     0.001\n",
      "  contains(soy)==True (1)                            0.000\n",
      "  contains(buttermilk)==True (1)                     0.000\n",
      "  contains(chipotle)==True (1)                       0.000\n",
      "  contains(starch)==True (1)                        -0.000\n",
      "  contains(empty)==True (1)                         -0.000\n",
      "  contains(translucent)==True (1)                    0.000\n",
      "  contains(bun)==True (1)                            0.000\n",
      "  contains(inevitably)==True (1)                    -0.000\n",
      "  contains(rd)==True (1)                             0.000\n",
      "  contains(dreamed)==True (1)                        0.000\n",
      "  contains(platter)==True (1)                        0.000\n",
      "  contains(ranch)==True (1)                         -0.000\n",
      "  contains(paprika)==True (1)                        0.000\n",
      "  contains(grandmother)==True (1)                    0.000\n",
      "  contains(dredged)==True (1)                        0.000\n",
      "  label is 'essays' (1)                                     -2.418\n",
      "  contains(’)==True (1)                                      0.767\n",
      "  contains(.”)==True (1)                                     0.597\n",
      "  contains(“)==True (1)                                      0.518\n",
      "  contains(u)==True (1)                                      0.451\n",
      "  contains(term)==True (1)                                   0.442\n",
      "  contains(—)==True (1)                                      0.413\n",
      "  contains(tom)==True (1)                                    0.385\n",
      "  contains(toward)==True (1)                                 0.369\n",
      "  contains(best)==True (1)                                   0.337\n",
      "  contains(year)==True (1)                                  -0.326\n",
      "  contains(crime)==True (1)                                  0.285\n",
      "  contains(almost)==True (1)                                 0.284\n",
      "  contains(want)==True (1)                                  -0.282\n",
      "  contains(new)==True (1)                                   -0.281\n",
      "  contains(change)==True (1)                                 0.270\n",
      "  contains(5)==True (1)                                      0.269\n",
      "  contains(old)==True (1)                                    0.268\n",
      "  contains(child)==True (1)                                  0.262\n",
      "  contains(good)==True (1)                                  -0.260\n",
      "  contains(time)==True (1)                                   0.255\n",
      "  contains(go)==True (1)                                     0.254\n",
      "  contains(joint)==True (1)                                  0.247\n",
      "  contains(fat)==True (1)                                    0.244\n",
      "  contains(effect)==True (1)                                 0.243\n",
      "  contains(created)==True (1)                                0.243\n",
      "  contains(said)==True (1)                                  -0.241\n",
      "  contains(name)==True (1)                                   0.238\n",
      "  contains(american)==True (1)                               0.236\n",
      "  contains(one)==True (1)                                    0.222\n",
      "  contains(lot)==True (1)                                   -0.215\n",
      "  contains(home)==True (1)                                  -0.214\n",
      "  contains(soul)==True (1)                                   0.212\n",
      "  contains(know)==True (1)                                   0.212\n",
      "  contains(never)==True (1)                                  0.211\n",
      "  contains(part)==True (1)                                  -0.211\n",
      "  contains(perfect)==True (1)                                0.206\n",
      "  contains(unless)==True (1)                                 0.199\n",
      "  contains(let)==True (1)                                   -0.198\n",
      "  contains(back)==True (1)                                  -0.191\n",
      "  contains(three)==True (1)                                  0.189\n",
      "  contains(10)==True (1)                                     0.187\n",
      "  contains(black)==True (1)                                  0.175\n",
      "  contains(kid)==True (1)                                    0.173\n",
      "  contains(open)==True (1)                                  -0.173\n",
      "  contains(butter)==True (1)                                 0.172\n",
      "  contains(hour)==True (1)                                   0.170\n",
      "  contains(ever)==True (1)                                   0.169\n",
      "  contains(say)==True (1)                                    0.167\n",
      "  contains(.’)==True (1)                                     0.165\n",
      "  contains(free)==True (1)                                   0.163\n",
      "  contains(made)==True (1)                                  -0.161\n",
      "  contains(actually)==True (1)                              -0.160\n",
      "  contains(better)==True (1)                                -0.158\n",
      "  contains(person)==True (1)                                -0.156\n",
      "  contains(need)==True (1)                                  -0.155\n",
      "  contains(much)==True (1)                                  -0.153\n",
      "  contains(form)==True (1)                                   0.153\n",
      "  contains(street)==True (1)                                 0.153\n",
      "  contains(asking)==True (1)                                 0.149\n",
      "  contains(using)==True (1)                                 -0.148\n",
      "  contains(whole)==True (1)                                  0.146\n",
      "  contains(oil)==True (1)                                    0.145\n",
      "  contains(adult)==True (1)                                  0.142\n",
      "  contains(list)==True (1)                                   0.141\n",
      "  contains(subtle)==True (1)                                 0.141\n",
      "  contains(exchange)==True (1)                               0.140\n",
      "  contains(take)==True (1)                                  -0.140\n",
      "  contains(like)==True (1)                                   0.139\n",
      "  contains(dad)==True (1)                                    0.138\n",
      "  contains(two)==True (1)                                    0.138\n",
      "  contains(secret)==True (1)                                 0.136\n",
      "  contains(boardwalk)==True (1)                              0.136\n",
      "  contains(spend)==True (1)                                  0.131\n",
      "  contains(bird)==True (1)                                   0.130\n",
      "  contains(sometimes)==True (1)                             -0.129\n",
      "  contains(come)==True (1)                                   0.129\n",
      "  contains(even)==True (1)                                  -0.126\n",
      "  contains(outer)==True (1)                                  0.125\n",
      "  contains(waffle)==True (1)                                 0.123\n",
      "  contains(thought)==True (1)                                0.121\n",
      "  contains(father)==True (1)                                 0.121\n",
      "  contains(house)==True (1)                                  0.121\n",
      "  contains(knife)==True (1)                                  0.121\n",
      "  contains(big)==True (1)                                   -0.121\n",
      "  contains(high)==True (1)                                   0.118\n",
      "  contains(enough)==True (1)                                -0.118\n",
      "  contains(”)==True (1)                                     -0.117\n",
      "  contains(.))==True (1)                                     0.116\n",
      "  contains(become)==True (1)                                -0.115\n",
      "  contains(le)==True (1)                                    -0.114\n",
      "  contains(‘)==True (1)                                      0.109\n",
      "  contains(pennsylvania)==True (1)                           0.107\n",
      "  contains(store)==True (1)                                  0.106\n",
      "  contains(something)==True (1)                             -0.103\n",
      "  contains().)==True (1)                                    -0.101\n",
      "  contains(word)==True (1)                                   0.101\n",
      "  contains(describe)==True (1)                               0.099\n",
      "  contains(central)==True (1)                                0.098\n",
      "  contains(columbia)==True (1)                               0.097\n",
      "  contains(line)==True (1)                                   0.096\n",
      "  contains(john)==True (1)                                   0.094\n",
      "  contains(style)==True (1)                                 -0.093\n",
      "  contains(8)==True (1)                                     -0.093\n",
      "  contains(roof)==True (1)                                   0.091\n",
      "  contains(virginia)==True (1)                               0.091\n",
      "  contains(quite)==True (1)                                 -0.091\n",
      "  contains(really)==True (1)                                 0.089\n",
      "  contains(right)==True (1)                                 -0.089\n",
      "  contains(adam)==True (1)                                   0.089\n",
      "  contains(month)==True (1)                                 -0.088\n",
      "  contains(white)==True (1)                                  0.087\n",
      "  contains(hard)==True (1)                                  -0.087\n",
      "  contains(important)==True (1)                             -0.086\n",
      "  contains(true)==True (1)                                  -0.086\n",
      "  contains(1971)==True (1)                                   0.084\n",
      "  contains(easy)==True (1)                                  -0.084\n",
      "  contains(seems)==True (1)                                 -0.083\n",
      "  contains(follows)==True (1)                                0.082\n",
      "  contains(thigh)==True (1)                                  0.082\n",
      "  contains(leave)==True (1)                                  0.081\n",
      "  contains(cornflakes)==True (1)                             0.080\n",
      "  contains(kitchen)==True (1)                                0.080\n",
      "  contains(later)==True (1)                                  0.080\n",
      "  contains(leg)==True (1)                                    0.080\n",
      "  contains(location)==True (1)                               0.079\n",
      "  contains(post)==True (1)                                   0.079\n",
      "  contains(favorite)==True (1)                              -0.079\n",
      "  contains(also)==True (1)                                   0.078\n",
      "  contains(resulting)==True (1)                              0.078\n",
      "  contains(probably)==True (1)                              -0.077\n",
      "  contains(rather)==True (1)                                -0.075\n",
      "  contains(pour)==True (1)                                   0.074\n",
      "  contains(everything)==True (1)                             0.074\n",
      "  contains(get)==True (1)                                    0.074\n",
      "  contains(found)==True (1)                                 -0.073\n",
      "  contains(anything)==True (1)                              -0.071\n",
      "  contains(sound)==True (1)                                  0.070\n",
      "  contains(paper)==True (1)                                  0.070\n",
      "  contains(agree)==True (1)                                  0.070\n",
      "  contains(richard)==True (1)                                0.069\n",
      "  contains(den)==True (1)                                    0.069\n",
      "  contains(hearty)==True (1)                                 0.069\n",
      "  contains(keep)==True (1)                                  -0.069\n",
      "  contains(alley)==True (1)                                  0.069\n",
      "  contains(age)==True (1)                                   -0.065\n",
      "  contains(miss)==True (1)                                   0.065\n",
      "  contains(basically)==True (1)                              0.065\n",
      "  contains(kind)==True (1)                                   0.064\n",
      "  contains(($)==True (1)                                     0.062\n",
      "  contains(believe)==True (1)                               -0.061\n",
      "  contains(fall)==True (1)                                  -0.060\n",
      "  contains(philippine)==True (1)                             0.059\n",
      "  contains(j)==True (1)                                      0.058\n",
      "  contains(teeth)==True (1)                                  0.057\n",
      "  contains(brand)==True (1)                                  0.057\n",
      "  contains(modern)==True (1)                                 0.057\n",
      "  contains(place)==True (1)                                 -0.056\n",
      "  contains(deal)==True (1)                                  -0.056\n",
      "  contains(stuff)==True (1)                                  0.054\n",
      "  contains(since)==True (1)                                 -0.054\n",
      "  contains(salt)==True (1)                                   0.054\n",
      "  contains(mac)==True (1)                                    0.054\n",
      "  contains(sell)==True (1)                                   0.053\n",
      "  contains(clean)==True (1)                                  0.053\n",
      "  contains(order)==True (1)                                 -0.053\n",
      "  contains(option)==True (1)                                 0.053\n",
      "  contains(,”)==True (1)                                     0.053\n",
      "  contains(begin)==True (1)                                  0.052\n",
      "  contains(long)==True (1)                                  -0.051\n",
      "  contains(classic)==True (1)                                0.051\n",
      "  contains(chicken)==True (1)                                0.050\n",
      "  contains(area)==True (1)                                  -0.050\n",
      "  contains(popular)==True (1)                               -0.050\n",
      "  contains(plate)==True (1)                                  0.050\n",
      "  contains(friday)==True (1)                                -0.048\n",
      "  contains(would)==True (1)                                  0.048\n",
      "  contains(pool)==True (1)                                   0.048\n",
      "  contains(c)==True (1)                                      0.048\n",
      "  contains(12)==True (1)                                    -0.048\n",
      "  contains(woman)==True (1)                                 -0.047\n",
      "  contains(95)==True (1)                                     0.047\n",
      "  contains(consistently)==True (1)                           0.047\n",
      "  contains(happy)==True (1)                                 -0.047\n",
      "  contains(half)==True (1)                                   0.046\n",
      "  contains(presentation)==True (1)                           0.046\n",
      "  contains(),)==True (1)                                    -0.045\n",
      "  contains(luxury)==True (1)                                 0.044\n",
      "  contains(turned)==True (1)                                -0.044\n",
      "  contains(excuse)==True (1)                                 0.044\n",
      "  contains(dedicated)==True (1)                              0.042\n",
      "  contains(paid)==True (1)                                   0.042\n",
      "  contains(thin)==True (1)                                   0.042\n",
      "  contains(wanted)==True (1)                                 0.041\n",
      "  contains(without)==True (1)                               -0.040\n",
      "  contains(used)==True (1)                                  -0.040\n",
      "  contains(another)==True (1)                               -0.040\n",
      "  contains(including)==True (1)                              0.039\n",
      "  contains(forget)==True (1)                                -0.039\n",
      "  contains(got)==True (1)                                    0.039\n",
      "  contains(bay)==True (1)                                    0.039\n",
      "  contains(pas)==True (1)                                    0.038\n",
      "  contains(south)==True (1)                                  0.038\n",
      "  contains(real)==True (1)                                  -0.037\n",
      "  contains(others)==True (1)                                -0.037\n",
      "  contains(side)==True (1)                                  -0.037\n",
      "  contains(eat)==True (1)                                    0.037\n",
      "  contains(buy)==True (1)                                    0.037\n",
      "  contains(maybe)==True (1)                                  0.035\n",
      "  contains(mean)==True (1)                                   0.035\n",
      "  contains(99)==True (1)                                     0.034\n",
      "  contains(offer)==True (1)                                  0.033\n",
      "  contains(twice)==True (1)                                  0.032\n",
      "  contains(circle)==True (1)                                 0.031\n",
      "  contains(nine)==True (1)                                   0.031\n",
      "  contains(run)==True (1)                                    0.030\n",
      "  contains(near)==True (1)                                  -0.030\n",
      "  contains(started)==True (1)                               -0.030\n",
      "  contains(pop)==True (1)                                    0.029\n",
      "  contains(pack)==True (1)                                   0.029\n",
      "  contains(getting)==True (1)                                0.029\n",
      "  contains(within)==True (1)                                -0.028\n",
      "  contains(music)==True (1)                                  0.026\n",
      "  contains(single)==True (1)                                -0.025\n",
      "  contains(east)==True (1)                                   0.025\n",
      "  contains(well)==True (1)                                  -0.025\n",
      "  contains(review)==True (1)                                 0.024\n",
      "  contains(14)==True (1)                                     0.024\n",
      "  contains(french)==True (1)                                 0.024\n",
      "  contains(call)==True (1)                                  -0.024\n",
      "  contains(whether)==True (1)                                0.023\n",
      "  contains(reveal)==True (1)                                 0.023\n",
      "  contains(tim)==True (1)                                    0.023\n",
      "  contains(self)==True (1)                                   0.023\n",
      "  contains(although)==True (1)                               0.023\n",
      "  contains(flash)==True (1)                                  0.022\n",
      "  contains(beauty)==True (1)                                -0.022\n",
      "  contains(heavy)==True (1)                                  0.021\n",
      "  contains(called)==True (1)                                -0.020\n",
      "  contains(care)==True (1)                                  -0.020\n",
      "  contains(offering)==True (1)                              -0.019\n",
      "  contains(account)==True (1)                               -0.017\n",
      "  contains(feel)==True (1)                                  -0.017\n",
      "  contains(everywhere)==True (1)                             0.017\n",
      "  contains(n)==True (1)                                      0.017\n",
      "  contains(competition)==True (1)                            0.017\n",
      "  contains(certain)==True (1)                               -0.016\n",
      "  contains(could)==True (1)                                 -0.016\n",
      "  contains(hand)==True (1)                                   0.016\n",
      "  contains(former)==True (1)                                -0.016\n",
      "  contains(r)==True (1)                                      0.016\n",
      "  contains(though)==True (1)                                -0.014\n",
      "  contains(fine)==True (1)                                  -0.012\n",
      "  contains(dark)==True (1)                                   0.012\n",
      "  contains(learned)==True (1)                               -0.012\n",
      "  contains(minute)==True (1)                                -0.011\n",
      "  contains(food)==True (1)                                   0.011\n",
      "  contains(owner)==True (1)                                  0.011\n",
      "  contains(deep)==True (1)                                  -0.010\n",
      "  contains(may)==True (1)                                   -0.010\n",
      "  contains(taken)==True (1)                                 -0.010\n",
      "  contains(glass)==True (1)                                  0.009\n",
      "  contains(double)==True (1)                                 0.009\n",
      "  contains(voice)==True (1)                                 -0.009\n",
      "  contains(common)==True (1)                                -0.008\n",
      "  contains(meat)==True (1)                                  -0.007\n",
      "  contains(!))==True (1)                                    -0.006\n",
      "  contains(country)==True (1)                                0.005\n",
      "  contains(thing)==True (1)                                  0.004\n",
      "  contains(every)==True (1)                                 -0.003\n",
      "  contains(driving)==True (1)                                0.002\n",
      "  contains(often)==True (1)                                 -0.000\n",
      "  contains(e)==True (1)                                              1.766\n",
      "  contains(“)==True (1)                                              1.441\n",
      "  contains(‘)==True (1)                                              1.095\n",
      "  contains(9)==True (1)                                              0.865\n",
      "  contains(child)==True (1)                                          0.863\n",
      "  contains(,”)==True (1)                                             0.859\n",
      "  contains(’)==True (1)                                              0.745\n",
      "  contains(finished)==True (1)                                       0.672\n",
      "  contains(age)==True (1)                                            0.604\n",
      "  contains(”)==True (1)                                              0.577\n",
      "  contains(bone)==True (1)                                           0.549\n",
      "  contains(u)==True (1)                                             -0.520\n",
      "  contains(best)==True (1)                                          -0.519\n",
      "  contains(.”)==True (1)                                             0.503\n",
      "  contains(margaret)==True (1)                                       0.488\n",
      "  contains(believe)==True (1)                                        0.477\n",
      "  contains(family)==True (1)                                         0.477\n",
      "  contains(get)==True (1)                                           -0.474\n",
      "  contains(14)==True (1)                                             0.436\n",
      "  label is 'books' (1)                                              -0.416\n",
      "  contains(like)==True (1)                                          -0.412\n",
      "  contains(review)==True (1)                                         0.410\n",
      "  contains(well)==True (1)                                           0.395\n",
      "  contains(5)==True (1)                                             -0.388\n",
      "  contains(crime)==True (1)                                          0.380\n",
      "  contains(friday)==True (1)                                         0.367\n",
      "  contains(bring)==True (1)                                          0.355\n",
      "  contains(old)==True (1)                                            0.335\n",
      "  contains(list)==True (1)                                           0.323\n",
      "  contains(american)==True (1)                                       0.317\n",
      "  contains(also)==True (1)                                           0.315\n",
      "  contains(post)==True (1)                                           0.314\n",
      "  contains(result)==True (1)                                        -0.308\n",
      "  contains(virginia)==True (1)                                       0.305\n",
      "  contains(store)==True (1)                                         -0.301\n",
      "  contains(end)==True (1)                                            0.299\n",
      "  contains(seems)==True (1)                                          0.296\n",
      "  contains(real)==True (1)                                          -0.292\n",
      "  contains(found)==True (1)                                          0.279\n",
      "  contains(created)==True (1)                                       -0.276\n",
      "  contains(others)==True (1)                                         0.270\n",
      "  contains(.))==True (1)                                             0.267\n",
      "  contains(open)==True (1)                                          -0.262\n",
      "  contains(something)==True (1)                                      0.262\n",
      "  contains(know)==True (1)                                           0.257\n",
      "  contains(called)==True (1)                                        -0.257\n",
      "  contains(change)==True (1)                                         0.250\n",
      "  contains(free)==True (1)                                          -0.248\n",
      "  contains(),)==True (1)                                             0.248\n",
      "  contains(since)==True (1)                                         -0.242\n",
      "  contains(top)==True (1)                                           -0.236\n",
      "  contains(time)==True (1)                                           0.235\n",
      "  contains(lot)==True (1)                                            0.224\n",
      "  contains(side)==True (1)                                          -0.224\n",
      "  contains(beer)==True (1)                                           0.223\n",
      "  contains(item)==True (1)                                           0.223\n",
      "  contains(big)==True (1)                                           -0.222\n",
      "  contains(place)==True (1)                                         -0.219\n",
      "  contains(popular)==True (1)                                       -0.217\n",
      "  contains(8)==True (1)                                             -0.215\n",
      "  contains(part)==True (1)                                           0.212\n",
      "  contains(le)==True (1)                                            -0.209\n",
      "  contains(bite)==True (1)                                           0.206\n",
      "  contains(former)==True (1)                                        -0.206\n",
      "  contains(made)==True (1)                                          -0.206\n",
      "  contains(form)==True (1)                                           0.206\n",
      "  contains(sure)==True (1)                                          -0.204\n",
      "  contains(tender)==True (1)                                         0.203\n",
      "  contains(matter)==True (1)                                         0.193\n",
      "  contains(year)==True (1)                                           0.191\n",
      "  contains(take)==True (1)                                          -0.190\n",
      "  contains(favorite)==True (1)                                      -0.189\n",
      "  contains(punishment)==True (1)                                     0.188\n",
      "  contains(deal)==True (1)                                          -0.185\n",
      "  contains(50)==True (1)                                             0.184\n",
      "  contains(instead)==True (1)                                        0.183\n",
      "  contains(25)==True (1)                                            -0.182\n",
      "  contains(modern)==True (1)                                        -0.181\n",
      "  contains(sandwich)==True (1)                                       0.180\n",
      "  contains(—)==True (1)                                             -0.177\n",
      "  contains(v)==True (1)                                             -0.176\n",
      "  contains(fully)==True (1)                                          0.176\n",
      "  contains(recipe)==True (1)                                        -0.175\n",
      "  contains(much)==True (1)                                          -0.172\n",
      "  contains(brand)==True (1)                                         -0.171\n",
      "  contains(commitment)==True (1)                                     0.170\n",
      "  contains(one)==True (1)                                           -0.170\n",
      "  contains(co)==True (1)                                            -0.170\n",
      "  contains(minute)==True (1)                                        -0.169\n",
      "  contains(often)==True (1)                                         -0.165\n",
      "  contains(12)==True (1)                                            -0.165\n",
      "  contains(started)==True (1)                                       -0.163\n",
      "  contains(could)==True (1)                                         -0.163\n",
      "  contains(let)==True (1)                                            0.161\n",
      "  contains(actually)==True (1)                                      -0.161\n",
      "  contains(head)==True (1)                                           0.160\n",
      "  contains(call)==True (1)                                          -0.158\n",
      "  contains(golden)==True (1)                                         0.158\n",
      "  contains(deep)==True (1)                                          -0.157\n",
      "  contains(traditional)==True (1)                                    0.157\n",
      "  contains(month)==True (1)                                          0.156\n",
      "  contains(john)==True (1)                                           0.156\n",
      "  contains(example)==True (1)                                       -0.156\n",
      "  contains(black)==True (1)                                         -0.155\n",
      "  contains(obsession)==True (1)                                      0.153\n",
      "  contains(two)==True (1)                                            0.152\n",
      "  contains(stock)==True (1)                                         -0.151\n",
      "  contains(r)==True (1)                                             -0.150\n",
      "  contains(offer)==True (1)                                          0.150\n",
      "  contains(cast)==True (1)                                          -0.149\n",
      "  contains(held)==True (1)                                          -0.148\n",
      "  contains(music)==True (1)                                         -0.148\n",
      "  contains(founder)==True (1)                                       -0.147\n",
      "  contains(fan)==True (1)                                           -0.147\n",
      "  contains(crack)==True (1)                                          0.146\n",
      "  contains(washington)==True (1)                                     0.145\n",
      "  contains(including)==True (1)                                      0.145\n",
      "  contains(father)==True (1)                                         0.139\n",
      "  contains(tim)==True (1)                                           -0.137\n",
      "  contains(three)==True (1)                                          0.137\n",
      "  contains(hold)==True (1)                                          -0.137\n",
      "  contains(available)==True (1)                                      0.136\n",
      "  contains(come)==True (1)                                           0.135\n",
      "  contains(street)==True (1)                                         0.134\n",
      "  contains(double)==True (1)                                         0.132\n",
      "  contains(version)==True (1)                                        0.130\n",
      "  contains(anything)==True (1)                                       0.128\n",
      "  contains(yard)==True (1)                                           0.125\n",
      "  contains(hot)==True (1)                                           -0.123\n",
      "  contains(try)==True (1)                                            0.123\n",
      "  contains(wing)==True (1)                                          -0.123\n",
      "  contains(paper)==True (1)                                         -0.122\n",
      "  contains(single)==True (1)                                        -0.122\n",
      "  contains(without)==True (1)                                       -0.120\n",
      "  contains(new)==True (1)                                            0.118\n",
      "  contains(go)==True (1)                                             0.118\n",
      "  contains(name)==True (1)                                          -0.117\n",
      "  contains(every)==True (1)                                         -0.116\n",
      "  contains(secret)==True (1)                                        -0.116\n",
      "  contains(high)==True (1)                                          -0.116\n",
      "  contains(offering)==True (1)                                      -0.116\n",
      "  contains(dad)==True (1)                                            0.116\n",
      "  contains(important)==True (1)                                     -0.115\n",
      "  contains(got)==True (1)                                           -0.113\n",
      "  contains(fresh)==True (1)                                         -0.112\n",
      "  contains(discover)==True (1)                                      -0.112\n",
      "  contains(white)==True (1)                                         -0.112\n",
      "  contains(everything)==True (1)                                    -0.110\n",
      "  contains(decade)==True (1)                                         0.110\n",
      "  contains(maybe)==True (1)                                          0.109\n",
      "  contains(finish)==True (1)                                        -0.108\n",
      "  contains(option)==True (1)                                        -0.108\n",
      "  contains(using)==True (1)                                         -0.108\n",
      "  contains(roll)==True (1)                                          -0.105\n",
      "  contains(owner)==True (1)                                         -0.105\n",
      "  contains(though)==True (1)                                         0.105\n",
      "  contains(tom)==True (1)                                           -0.105\n",
      "  contains(adult)==True (1)                                          0.105\n",
      "  contains(kid)==True (1)                                           -0.102\n",
      "  contains(shard)==True (1)                                          0.102\n",
      "  contains(getting)==True (1)                                       -0.101\n",
      "  contains(line)==True (1)                                           0.099\n",
      "  contains(10)==True (1)                                            -0.098\n",
      "  contains(word)==True (1)                                           0.098\n",
      "  contains(happy)==True (1)                                         -0.097\n",
      "  contains(area)==True (1)                                          -0.097\n",
      "  contains(french)==True (1)                                        -0.097\n",
      "  contains(creator)==True (1)                                       -0.097\n",
      "  contains(build)==True (1)                                         -0.096\n",
      "  contains(sound)==True (1)                                          0.094\n",
      "  contains(cooking)==True (1)                                       -0.094\n",
      "  contains(whether)==True (1)                                        0.091\n",
      "  contains(method)==True (1)                                        -0.091\n",
      "  contains(house)==True (1)                                         -0.090\n",
      "  contains(trick)==True (1)                                         -0.090\n",
      "  contains(fancy)==True (1)                                          0.089\n",
      "  contains(low)==True (1)                                           -0.088\n",
      "  contains(sold)==True (1)                                           0.088\n",
      "  contains(run)==True (1)                                            0.086\n",
      "  contains(probably)==True (1)                                       0.085\n",
      "  contains(whole)==True (1)                                         -0.084\n",
      "  contains(wolf)==True (1)                                           0.084\n",
      "  contains(even)==True (1)                                          -0.083\n",
      "  contains(developed)==True (1)                                     -0.083\n",
      "  contains().)==True (1)                                            -0.083\n",
      "  contains(newest)==True (1)                                        -0.082\n",
      "  contains(near)==True (1)                                           0.082\n",
      "  contains(pop)==True (1)                                           -0.081\n",
      "  contains(really)==True (1)                                         0.081\n",
      "  contains(enough)==True (1)                                        -0.081\n",
      "  contains(com)==True (1)                                           -0.080\n",
      "  contains(named)==True (1)                                         -0.080\n",
      "  contains(dark)==True (1)                                          -0.080\n",
      "  contains(respect)==True (1)                                       -0.080\n",
      "  contains(voice)==True (1)                                          0.079\n",
      "  contains(nine)==True (1)                                           0.079\n",
      "  contains(effect)==True (1)                                        -0.078\n",
      "  contains(fast)==True (1)                                          -0.078\n",
      "  contains(taking)==True (1)                                        -0.078\n",
      "  contains(competition)==True (1)                                   -0.077\n",
      "  contains(prevent)==True (1)                                       -0.077\n",
      "  contains(used)==True (1)                                          -0.077\n",
      "  contains(michel)==True (1)                                         0.077\n",
      "  contains(taylor)==True (1)                                         0.077\n",
      "  contains(sea)==True (1)                                           -0.077\n",
      "  contains(operation)==True (1)                                     -0.076\n",
      "  contains(teeth)==True (1)                                         -0.076\n",
      "  contains(fine)==True (1)                                          -0.075\n",
      "  contains(dig)==True (1)                                            0.075\n",
      "  contains(butterfly)==True (1)                                      0.075\n",
      "  contains(keep)==True (1)                                           0.075\n",
      "  contains(never)==True (1)                                         -0.074\n",
      "  contains(plate)==True (1)                                         -0.074\n",
      "  contains(stuff)==True (1)                                         -0.074\n",
      "  contains(bay)==True (1)                                           -0.073\n",
      "  contains(style)==True (1)                                          0.073\n",
      "  contains(follows)==True (1)                                        0.072\n",
      "  contains(person)==True (1)                                         0.072\n",
      "  contains(begin)==True (1)                                          0.071\n",
      "  contains(spend)==True (1)                                         -0.071\n",
      "  contains(long)==True (1)                                           0.070\n",
      "  contains(standard)==True (1)                                       0.069\n",
      "  contains(alike)==True (1)                                         -0.069\n",
      "  contains(arrives)==True (1)                                       -0.069\n",
      "  contains(signature)==True (1)                                      0.068\n",
      "  contains(adam)==True (1)                                          -0.068\n",
      "  contains(snack)==True (1)                                         -0.068\n",
      "  contains(juice)==True (1)                                         -0.067\n",
      "  contains(leave)==True (1)                                          0.066\n",
      "  contains(half)==True (1)                                           0.066\n",
      "  contains(variety)==True (1)                                       -0.066\n",
      "  contains(south)==True (1)                                         -0.065\n",
      "  contains(chicken)==True (1)                                       -0.065\n",
      "  contains(blend)==True (1)                                         -0.065\n",
      "  contains(longer)==True (1)                                        -0.064\n",
      "  contains(needed)==True (1)                                        -0.063\n",
      "  contains(set)==True (1)                                            0.063\n",
      "  contains(fish)==True (1)                                           0.063\n",
      "  contains(kitchen)==True (1)                                       -0.063\n",
      "  contains(eric)==True (1)                                          -0.063\n",
      "  contains(perfectly)==True (1)                                      0.062\n",
      "  contains(bird)==True (1)                                          -0.062\n",
      "  contains(leg)==True (1)                                           -0.062\n",
      "  contains(creating)==True (1)                                      -0.061\n",
      "  contains(fat)==True (1)                                           -0.061\n",
      "  contains(ordering)==True (1)                                       0.060\n",
      "  contains(central)==True (1)                                       -0.060\n",
      "  contains(thought)==True (1)                                       -0.060\n",
      "  contains(hill)==True (1)                                           0.059\n",
      "  contains(japanese)==True (1)                                      -0.058\n",
      "  contains(biscuit)==True (1)                                        0.058\n",
      "  contains(woman)==True (1)                                          0.057\n",
      "  contains(crushed)==True (1)                                        0.057\n",
      "  contains(enjoyed)==True (1)                                       -0.055\n",
      "  contains(describe)==True (1)                                      -0.055\n",
      "  contains(reveal)==True (1)                                         0.055\n",
      "  contains(dedicated)==True (1)                                     -0.055\n",
      "  contains(bag)==True (1)                                           -0.055\n",
      "  contains(unless)==True (1)                                         0.054\n",
      "  contains(occasional)==True (1)                                     0.054\n",
      "  contains(beauty)==True (1)                                         0.053\n",
      "  contains(fly)==True (1)                                            0.053\n",
      "  contains(richard)==True (1)                                       -0.053\n",
      "  contains(24)==True (1)                                            -0.053\n",
      "  contains(say)==True (1)                                            0.053\n",
      "  contains(simply)==True (1)                                        -0.052\n",
      "  contains(rose)==True (1)                                          -0.052\n",
      "  contains(toward)==True (1)                                        -0.052\n",
      "  contains(cart)==True (1)                                           0.052\n",
      "  contains(however)==True (1)                                        0.052\n",
      "  contains(beef)==True (1)                                           0.051\n",
      "  contains(country)==True (1)                                       -0.051\n",
      "  contains(home)==True (1)                                           0.051\n",
      "  contains(n)==True (1)                                             -0.051\n",
      "  contains(self)==True (1)                                           0.051\n",
      "  contains(thin)==True (1)                                          -0.051\n",
      "  contains(den)==True (1)                                           -0.051\n",
      "  contains(back)==True (1)                                          -0.050\n",
      "  contains(j)==True (1)                                             -0.050\n",
      "  contains(pas)==True (1)                                           -0.050\n",
      "  contains(buy)==True (1)                                            0.050\n",
      "  contains(grab)==True (1)                                          -0.049\n",
      "  contains(would)==True (1)                                          0.049\n",
      "  contains(suit)==True (1)                                          -0.049\n",
      "  contains(credit)==True (1)                                         0.049\n",
      "  contains(may)==True (1)                                           -0.049\n",
      "  contains(piece)==True (1)                                          0.049\n",
      "  contains(waste)==True (1)                                         -0.049\n",
      "  contains(dinner)==True (1)                                         0.048\n",
      "  contains(later)==True (1)                                         -0.048\n",
      "  contains(throughout)==True (1)                                     0.048\n",
      "  contains(consciousness)==True (1)                                  0.048\n",
      "  contains(frequently)==True (1)                                     0.047\n",
      "  contains(leaf)==True (1)                                          -0.047\n",
      "  contains(sell)==True (1)                                          -0.047\n",
      "  contains(allow)==True (1)                                         -0.047\n",
      "  contains(essentially)==True (1)                                   -0.047\n",
      "  contains(crystal)==True (1)                                        0.046\n",
      "  contains(sometimes)==True (1)                                      0.046\n",
      "  contains(wanted)==True (1)                                        -0.045\n",
      "  contains(classic)==True (1)                                        0.044\n",
      "  contains(warm)==True (1)                                           0.044\n",
      "  contains(luxury)==True (1)                                        -0.043\n",
      "  contains(attached)==True (1)                                      -0.043\n",
      "  contains(generous)==True (1)                                       0.043\n",
      "  contains(paid)==True (1)                                          -0.043\n",
      "  contains(spin)==True (1)                                          -0.042\n",
      "  contains(hour)==True (1)                                          -0.042\n",
      "  contains(accompanying)==True (1)                                   0.042\n",
      "  contains(passion)==True (1)                                       -0.042\n",
      "  contains(testing)==True (1)                                       -0.041\n",
      "  contains(learned)==True (1)                                        0.041\n",
      "  contains(inspired)==True (1)                                      -0.041\n",
      "  contains(true)==True (1)                                          -0.041\n",
      "  contains(ever)==True (1)                                          -0.040\n",
      "  contains(element)==True (1)                                       -0.040\n",
      "  contains(spent)==True (1)                                          0.040\n",
      "  contains(iron)==True (1)                                          -0.040\n",
      "  contains(local)==True (1)                                          0.039\n",
      "  contains(twice)==True (1)                                         -0.039\n",
      "  contains(asking)==True (1)                                         0.039\n",
      "  contains(stool)==True (1)                                         -0.039\n",
      "  contains(excuse)==True (1)                                        -0.039\n",
      "  contains(seasoned)==True (1)                                       0.039\n",
      "  contains(agree)==True (1)                                          0.039\n",
      "  contains(formula)==True (1)                                       -0.038\n",
      "  contains(better)==True (1)                                         0.038\n",
      "  contains(exchange)==True (1)                                      -0.037\n",
      "  contains(onto)==True (1)                                           0.037\n",
      "  contains(east)==True (1)                                          -0.037\n",
      "  contains(planning)==True (1)                                       0.037\n",
      "  contains(clean)==True (1)                                         -0.037\n",
      "  contains(exact)==True (1)                                          0.036\n",
      "  contains(farce)==True (1)                                          0.036\n",
      "  contains(want)==True (1)                                          -0.036\n",
      "  contains(right)==True (1)                                         -0.036\n",
      "  contains(pain)==True (1)                                          -0.035\n",
      "  contains(hard)==True (1)                                           0.035\n",
      "  contains(atop)==True (1)                                           0.035\n",
      "  contains(become)==True (1)                                        -0.035\n",
      "  contains(another)==True (1)                                        0.034\n",
      "  contains(c)==True (1)                                             -0.034\n",
      "  contains(cut)==True (1)                                           -0.034\n",
      "  contains(almost)==True (1)                                         0.034\n",
      "  contains(flash)==True (1)                                          0.033\n",
      "  contains(bar)==True (1)                                           -0.032\n",
      "  contains(finger)==True (1)                                        -0.032\n",
      "  contains(training)==True (1)                                      -0.032\n",
      "  contains(food)==True (1)                                           0.031\n",
      "  contains(common)==True (1)                                         0.031\n",
      "  contains(care)==True (1)                                          -0.030\n",
      "  contains(pressure)==True (1)                                      -0.030\n",
      "  contains(restaurant)==True (1)                                    -0.030\n",
      "  contains(chef)==True (1)                                          -0.030\n",
      "  contains(dose)==True (1)                                           0.030\n",
      "  contains(butter)==True (1)                                         0.030\n",
      "  contains(everywhere)==True (1)                                    -0.030\n",
      "  contains(variation)==True (1)                                     -0.029\n",
      "  contains(lily)==True (1)                                           0.029\n",
      "  contains(ask)==True (1)                                            0.029\n",
      "  contains(stuffed)==True (1)                                       -0.029\n",
      "  contains(within)==True (1)                                        -0.029\n",
      "  contains(forget)==True (1)                                         0.028\n",
      "  contains(surprising)==True (1)                                     0.028\n",
      "  contains(combination)==True (1)                                   -0.028\n",
      "  contains(.’)==True (1)                                             0.028\n",
      "  contains(taken)==True (1)                                          0.028\n",
      "  contains(inevitably)==True (1)                                     0.027\n",
      "  contains(heavy)==True (1)                                          0.027\n",
      "  contains(slightly)==True (1)                                       0.027\n",
      "  contains(southern)==True (1)                                       0.027\n",
      "  contains(topped)==True (1)                                        -0.027\n",
      "  contains(presentation)==True (1)                                  -0.026\n",
      "  contains(rather)==True (1)                                        -0.026\n",
      "  contains(combine)==True (1)                                        0.026\n",
      "  contains(base)==True (1)                                          -0.026\n",
      "  contains(frank)==True (1)                                         -0.025\n",
      "  contains(process)==True (1)                                       -0.025\n",
      "  contains(chain)==True (1)                                         -0.025\n",
      "  contains(plain)==True (1)                                         -0.025\n",
      "  contains(circle)==True (1)                                         0.025\n",
      "  contains(remains)==True (1)                                       -0.025\n",
      "  contains(strip)==True (1)                                         -0.024\n",
      "  contains(spring)==True (1)                                         0.024\n",
      "  contains(hank)==True (1)                                           0.024\n",
      "  contains(dennis)==True (1)                                         0.024\n",
      "  contains(yield)==True (1)                                         -0.024\n",
      "  contains(although)==True (1)                                      -0.024\n",
      "  contains(($)==True (1)                                             0.023\n",
      "  contains(bread)==True (1)                                          0.023\n",
      "  contains(shore)==True (1)                                          0.023\n",
      "  contains(bucket)==True (1)                                        -0.023\n",
      "  contains(protein)==True (1)                                        0.022\n",
      "  contains(skin)==True (1)                                           0.022\n",
      "  contains(gauche)==True (1)                                         0.022\n",
      "  contains(addictive)==True (1)                                     -0.022\n",
      "  contains(tartare)==True (1)                                        0.022\n",
      "  contains(plus)==True (1)                                           0.022\n",
      "  contains(picnic)==True (1)                                        -0.022\n",
      "  contains(!))==True (1)                                             0.022\n",
      "  contains(empty)==True (1)                                          0.022\n",
      "  contains(feel)==True (1)                                          -0.021\n",
      "  contains(ala)==True (1)                                            0.021\n",
      "  contains(baltimore)==True (1)                                     -0.021\n",
      "  contains(ear)==True (1)                                           -0.020\n",
      "  contains(steak)==True (1)                                          0.020\n",
      "  contains(stick)==True (1)                                          0.020\n",
      "  contains(cooper)==True (1)                                         0.020\n",
      "  contains(turned)==True (1)                                         0.020\n",
      "  contains(good)==True (1)                                          -0.020\n",
      "  contains(count)==True (1)                                         -0.020\n",
      "  contains(satisfying)==True (1)                                     0.019\n",
      "  contains(hand)==True (1)                                          -0.018\n",
      "  contains(degree)==True (1)                                        -0.018\n",
      "  contains(salt)==True (1)                                          -0.018\n",
      "  contains(perfect)==True (1)                                       -0.018\n",
      "  contains(rendered)==True (1)                                       0.018\n",
      "  contains(abbott)==True (1)                                         0.018\n",
      "  contains(clerk)==True (1)                                          0.017\n",
      "  contains(certain)==True (1)                                        0.017\n",
      "  contains(marriage)==True (1)                                       0.017\n",
      "  contains(need)==True (1)                                          -0.017\n",
      "  contains(silver)==True (1)                                         0.016\n",
      "  contains(distance)==True (1)                                       0.015\n",
      "  contains(resist)==True (1)                                        -0.015\n",
      "  contains(romeo)==True (1)                                          0.015\n",
      "  contains(breath)==True (1)                                         0.015\n",
      "  contains(thing)==True (1)                                          0.015\n",
      "  contains(japan)==True (1)                                         -0.015\n",
      "  contains(99)==True (1)                                            -0.015\n",
      "  contains(meal)==True (1)                                           0.015\n",
      "  contains(menu)==True (1)                                          -0.014\n",
      "  contains(basket)==True (1)                                         0.014\n",
      "  contains(soul)==True (1)                                          -0.014\n",
      "  contains(pack)==True (1)                                          -0.014\n",
      "  contains(roof)==True (1)                                           0.014\n",
      "  contains(spice)==True (1)                                         -0.014\n",
      "  contains(accused)==True (1)                                        0.013\n",
      "  contains(put)==True (1)                                            0.013\n",
      "  contains(ray)==True (1)                                            0.013\n",
      "  contains(2002)==True (1)                                          -0.013\n",
      "  contains(knife)==True (1)                                         -0.013\n",
      "  contains(extremely)==True (1)                                      0.013\n",
      "  contains(forgotten)==True (1)                                      0.013\n",
      "  contains(memorable)==True (1)                                     -0.012\n",
      "  contains(order)==True (1)                                          0.012\n",
      "  contains(tending)==True (1)                                        0.012\n",
      "  contains(shine)==True (1)                                         -0.012\n",
      "  contains(virtue)==True (1)                                        -0.012\n",
      "  contains(fall)==True (1)                                           0.012\n",
      "  contains(flesh)==True (1)                                          0.012\n",
      "  contains(.,)==True (1)                                            -0.012\n",
      "  contains(miss)==True (1)                                          -0.012\n",
      "  contains(roasted)==True (1)                                       -0.012\n",
      "  contains(deeply)==True (1)                                         0.012\n",
      "  contains(korean)==True (1)                                         0.011\n",
      "  contains(dignity)==True (1)                                        0.011\n",
      "  contains(appreciate)==True (1)                                    -0.011\n",
      "  contains(basically)==True (1)                                      0.011\n",
      "  contains(stacked)==True (1)                                       -0.011\n",
      "  contains(taste)==True (1)                                          0.011\n",
      "  contains(fried)==True (1)                                         -0.011\n",
      "  contains(layer)==True (1)                                         -0.010\n",
      "  contains(easy)==True (1)                                           0.010\n",
      "  contains(size)==True (1)                                           0.010\n",
      "  contains(wonderfully)==True (1)                                    0.010\n",
      "  contains(pan)==True (1)                                           -0.010\n",
      "  contains(kind)==True (1)                                           0.010\n",
      "  contains(morgan)==True (1)                                         0.010\n",
      "  contains(merit)==True (1)                                          0.010\n",
      "  contains(capitol)==True (1)                                        0.009\n",
      "  contains(treated)==True (1)                                       -0.009\n",
      "  contains(connecticut)==True (1)                                    0.009\n",
      "  contains(dipping)==True (1)                                        0.009\n",
      "  contains(account)==True (1)                                        0.009\n",
      "  contains(handle)==True (1)                                        -0.009\n",
      "  contains(detail)==True (1)                                         0.009\n",
      "  contains(sole)==True (1)                                           0.009\n",
      "  contains(filling)==True (1)                                       -0.009\n",
      "  contains(upscale)==True (1)                                        0.008\n",
      "  contains(bum)==True (1)                                            0.008\n",
      "  contains(said)==True (1)                                          -0.008\n",
      "  contains(st)==True (1)                                             0.008\n",
      "  contains(reserve)==True (1)                                        0.008\n",
      "  contains(appetite)==True (1)                                       0.008\n",
      "  contains(ubiquitous)==True (1)                                     0.008\n",
      "  contains(commonly)==True (1)                                       0.008\n",
      "  contains(involved)==True (1)                                       0.008\n",
      "  contains(smack)==True (1)                                          0.007\n",
      "  contains(district)==True (1)                                      -0.007\n",
      "  contains(primary)==True (1)                                       -0.007\n",
      "  contains(crumb)==True (1)                                          0.007\n",
      "  contains(served)==True (1)                                        -0.007\n",
      "  contains(exterior)==True (1)                                       0.007\n",
      "  contains(founded)==True (1)                                       -0.006\n",
      "  contains(korea)==True (1)                                          0.006\n",
      "  contains(ultra)==True (1)                                         -0.005\n",
      "  contains(pleasure)==True (1)                                      -0.005\n",
      "  contains(rob)==True (1)                                           -0.005\n",
      "  contains(mean)==True (1)                                           0.005\n",
      "  contains(estate)==True (1)                                         0.005\n",
      "  contains(portion)==True (1)                                       -0.004\n",
      "  contains(ranch)==True (1)                                          0.004\n",
      "  contains(honey)==True (1)                                         -0.004\n",
      "  contains(glass)==True (1)                                         -0.004\n",
      "  contains(pony)==True (1)                                           0.004\n",
      "  contains(sink)==True (1)                                          -0.004\n",
      "  contains(lunch)==True (1)                                         -0.004\n",
      "  contains(accent)==True (1)                                         0.004\n",
      "  contains(bath)==True (1)                                          -0.004\n",
      "  contains(argue)==True (1)                                         -0.004\n",
      "  contains(breast)==True (1)                                         0.004\n",
      "  contains(favor)==True (1)                                         -0.003\n",
      "  contains(emerge)==True (1)                                         0.003\n",
      "  contains(maryland)==True (1)                                       0.003\n",
      "  contains(pour)==True (1)                                           0.003\n",
      "  contains(quite)==True (1)                                         -0.003\n",
      "  contains(term)==True (1)                                           0.003\n",
      "  contains(ditch)==True (1)                                          0.003\n",
      "  contains(proper)==True (1)                                         0.002\n",
      "  contains(eat)==True (1)                                            0.002\n",
      "  contains(crunch)==True (1)                                         0.001\n",
      "  contains(technically)==True (1)                                    0.001\n",
      "  contains(lends)==True (1)                                          0.001\n",
      "  contains(famed)==True (1)                                          0.001\n",
      "  contains(chewing)==True (1)                                        0.001\n",
      "  contains(leftover)==True (1)                                      -0.001\n",
      "  contains(preparation)==True (1)                                    0.000\n",
      "  contains(driving)==True (1)                                        0.000\n",
      "  contains(georgia)==True (1)                                        0.000\n",
      "  contains(guilty)==True (1)                                         0.000\n",
      "  label is 'do_it_yourself' (1)                                             -1.957\n",
      "  contains(’)==True (1)                                                      0.986\n",
      "  contains().)==True (1)                                                     0.984\n",
      "  contains(need)==True (1)                                                   0.850\n",
      "  contains(like)==True (1)                                                   0.654\n",
      "  contains(using)==True (1)                                                  0.639\n",
      "  contains(wanted)==True (1)                                                 0.515\n",
      "  contains(year)==True (1)                                                  -0.476\n",
      "  contains(started)==True (1)                                                0.466\n",
      "  contains(sure)==True (1)                                                   0.453\n",
      "  contains(piece)==True (1)                                                  0.445\n",
      "  contains(although)==True (1)                                               0.430\n",
      "  contains(every)==True (1)                                                  0.388\n",
      "  contains(creating)==True (1)                                               0.364\n",
      "  contains(account)==True (1)                                                0.355\n",
      "  contains(know)==True (1)                                                  -0.338\n",
      "  contains(offer)==True (1)                                                  0.333\n",
      "  contains(place)==True (1)                                                  0.329\n",
      "  contains(build)==True (1)                                                  0.326\n",
      "  contains(r)==True (1)                                                     -0.304\n",
      "  contains(head)==True (1)                                                   0.302\n",
      "  contains(thing)==True (1)                                                  0.301\n",
      "  contains(used)==True (1)                                                   0.297\n",
      "  contains(9)==True (1)                                                     -0.287\n",
      "  contains(”)==True (1)                                                     -0.276\n",
      "  contains(near)==True (1)                                                   0.269\n",
      "  contains(hand)==True (1)                                                   0.267\n",
      "  contains(well)==True (1)                                                   0.260\n",
      "  contains(really)==True (1)                                                -0.256\n",
      "  contains(right)==True (1)                                                 -0.250\n",
      "  contains(spent)==True (1)                                                  0.248\n",
      "  contains(finger)==True (1)                                                 0.248\n",
      "  contains(option)==True (1)                                                 0.245\n",
      "  contains(low)==True (1)                                                    0.245\n",
      "  contains(cut)==True (1)                                                    0.244\n",
      "  contains(quite)==True (1)                                                 -0.244\n",
      "  contains(change)==True (1)                                                -0.238\n",
      "  contains(.))==True (1)                                                    -0.238\n",
      "  contains(pop)==True (1)                                                    0.238\n",
      "  contains(hard)==True (1)                                                  -0.236\n",
      "  contains(try)==True (1)                                                    0.234\n",
      "  contains(found)==True (1)                                                  0.230\n",
      "  contains(run)==True (1)                                                    0.229\n",
      "  contains(also)==True (1)                                                   0.226\n",
      "  contains(fresh)==True (1)                                                  0.225\n",
      "  contains(le)==True (1)                                                    -0.224\n",
      "  contains(stuff)==True (1)                                                 -0.223\n",
      "  contains(mean)==True (1)                                                  -0.221\n",
      "  contains(trick)==True (1)                                                 -0.219\n",
      "  contains(john)==True (1)                                                  -0.218\n",
      "  contains(seems)==True (1)                                                 -0.217\n",
      "  contains(probably)==True (1)                                              -0.209\n",
      "  contains(never)==True (1)                                                 -0.208\n",
      "  contains(best)==True (1)                                                  -0.207\n",
      "  contains(v)==True (1)                                                     -0.200\n",
      "  contains(forget)==True (1)                                                 0.199\n",
      "  contains(method)==True (1)                                                 0.198\n",
      "  contains(take)==True (1)                                                  -0.196\n",
      "  contains(part)==True (1)                                                   0.193\n",
      "  contains(important)==True (1)                                              0.191\n",
      "  contains(almost)==True (1)                                                -0.190\n",
      "  contains(food)==True (1)                                                  -0.188\n",
      "  contains(lot)==True (1)                                                   -0.187\n",
      "  contains(minute)==True (1)                                                 0.186\n",
      "  contains(‘)==True (1)                                                     -0.185\n",
      "  contains(foam)==True (1)                                                   0.184\n",
      "  contains(thought)==True (1)                                                0.183\n",
      "  contains(ask)==True (1)                                                    0.177\n",
      "  contains(result)==True (1)                                                -0.176\n",
      "  contains(say)==True (1)                                                   -0.176\n",
      "  contains(mount)==True (1)                                                  0.176\n",
      "  contains(5)==True (1)                                                      0.176\n",
      "  contains(butterfly)==True (1)                                              0.175\n",
      "  contains(could)==True (1)                                                 -0.173\n",
      "  contains(classic)==True (1)                                               -0.170\n",
      "  contains(handle)==True (1)                                                 0.169\n",
      "  contains(menu)==True (1)                                                   0.168\n",
      "  contains(former)==True (1)                                                -0.167\n",
      "  contains(local)==True (1)                                                  0.166\n",
      "  contains(pour)==True (1)                                                   0.163\n",
      "  contains(clean)==True (1)                                                  0.163\n",
      "  contains(top)==True (1)                                                    0.161\n",
      "  contains(south)==True (1)                                                 -0.161\n",
      "  contains(review)==True (1)                                                -0.160\n",
      "  contains(miss)==True (1)                                                  -0.156\n",
      "  contains(base)==True (1)                                                   0.153\n",
      "  contains(detail)==True (1)                                                 0.152\n",
      "  contains(stock)==True (1)                                                  0.151\n",
      "  contains(finished)==True (1)                                               0.150\n",
      "  contains(resulting)==True (1)                                              0.149\n",
      "  contains(single)==True (1)                                                 0.147\n",
      "  contains(!))==True (1)                                                    -0.146\n",
      "  contains(recipe)==True (1)                                                -0.145\n",
      "  contains(woman)==True (1)                                                 -0.144\n",
      "  contains(free)==True (1)                                                   0.144\n",
      "  contains(said)==True (1)                                                  -0.144\n",
      "  contains(outer)==True (1)                                                  0.144\n",
      "  contains(begin)==True (1)                                                  0.141\n",
      "  contains(99)==True (1)                                                    -0.140\n",
      "  contains(taking)==True (1)                                                 0.140\n",
      "  contains(double)==True (1)                                                -0.138\n",
      "  contains(8)==True (1)                                                      0.138\n",
      "  contains(fall)==True (1)                                                   0.138\n",
      "  contains(may)==True (1)                                                    0.138\n",
      "  contains(fish)==True (1)                                                   0.137\n",
      "  contains(feel)==True (1)                                                   0.134\n",
      "  contains(called)==True (1)                                                -0.133\n",
      "  contains(everything)==True (1)                                             0.133\n",
      "  contains(temperature)==True (1)                                            0.133\n",
      "  contains(co)==True (1)                                                     0.133\n",
      "  contains(month)==True (1)                                                 -0.133\n",
      "  contains(hour)==True (1)                                                   0.131\n",
      "  contains(driving)==True (1)                                               -0.129\n",
      "  contains(made)==True (1)                                                   0.129\n",
      "  contains(roll)==True (1)                                                  -0.129\n",
      "  contains(item)==True (1)                                                   0.128\n",
      "  contains(instead)==True (1)                                                0.126\n",
      "  contains(central)==True (1)                                                0.126\n",
      "  contains(50)==True (1)                                                    -0.125\n",
      "  contains(creator)==True (1)                                               -0.124\n",
      "  contains(created)==True (1)                                               -0.123\n",
      "  contains(put)==True (1)                                                    0.122\n",
      "  contains(strip)==True (1)                                                  0.122\n",
      "  contains(black)==True (1)                                                 -0.121\n",
      "  contains(founder)==True (1)                                               -0.120\n",
      "  contains(asking)==True (1)                                                -0.118\n",
      "  contains(become)==True (1)                                                -0.118\n",
      "  contains(post)==True (1)                                                   0.118\n",
      "  contains(fine)==True (1)                                                   0.117\n",
      "  contains(location)==True (1)                                               0.117\n",
      "  contains(needed)==True (1)                                                -0.116\n",
      "  contains(fancy)==True (1)                                                 -0.116\n",
      "  contains(size)==True (1)                                                   0.115\n",
      "  contains(u)==True (1)                                                     -0.113\n",
      "  contains(order)==True (1)                                                  0.113\n",
      "  contains(age)==True (1)                                                    0.112\n",
      "  contains(simply)==True (1)                                                -0.112\n",
      "  contains(let)==True (1)                                                   -0.111\n",
      "  contains(believe)==True (1)                                               -0.111\n",
      "  contains(learned)==True (1)                                               -0.110\n",
      "  contains(tim)==True (1)                                                   -0.110\n",
      "  contains(easy)==True (1)                                                  -0.109\n",
      "  contains(kind)==True (1)                                                  -0.109\n",
      "  contains(extremely)==True (1)                                              0.109\n",
      "  contains(owner)==True (1)                                                 -0.109\n",
      "  contains(district)==True (1)                                               0.108\n",
      "  contains(sound)==True (1)                                                  0.108\n",
      "  contains(attached)==True (1)                                               0.107\n",
      "  contains(training)==True (1)                                              -0.105\n",
      "  contains(southern)==True (1)                                               0.105\n",
      "  contains(thin)==True (1)                                                   0.104\n",
      "  contains(brand)==True (1)                                                 -0.103\n",
      "  contains(prevent)==True (1)                                                0.102\n",
      "  contains(come)==True (1)                                                   0.102\n",
      "  contains(testing)==True (1)                                               -0.101\n",
      "  contains(portion)==True (1)                                                0.101\n",
      "  contains(dark)==True (1)                                                  -0.100\n",
      "  contains(10)==True (1)                                                    -0.100\n",
      "  contains(finish)==True (1)                                                -0.100\n",
      "  contains(care)==True (1)                                                  -0.100\n",
      "  contains(st)==True (1)                                                     0.100\n",
      "  contains(later)==True (1)                                                 -0.099\n",
      "  contains(sometimes)==True (1)                                             -0.099\n",
      "  contains(onto)==True (1)                                                   0.097\n",
      "  contains(three)==True (1)                                                 -0.097\n",
      "  contains(bucket)==True (1)                                                 0.097\n",
      "  contains(self)==True (1)                                                  -0.095\n",
      "  contains(want)==True (1)                                                  -0.095\n",
      "  contains(leg)==True (1)                                                    0.095\n",
      "  contains(restaurant)==True (1)                                            -0.094\n",
      "  contains(go)==True (1)                                                     0.094\n",
      "  contains(friday)==True (1)                                                 0.094\n",
      "  contains(voice)==True (1)                                                 -0.093\n",
      "  contains(music)==True (1)                                                  0.093\n",
      "  contains(plus)==True (1)                                                  -0.093\n",
      "  contains(degree)==True (1)                                                 0.092\n",
      "  contains(enjoyed)==True (1)                                               -0.092\n",
      "  contains(effect)==True (1)                                                -0.092\n",
      "  contains(leaf)==True (1)                                                   0.092\n",
      "  contains(wing)==True (1)                                                   0.092\n",
      "  contains(others)==True (1)                                                -0.091\n",
      "  contains(,”)==True (1)                                                     0.090\n",
      "  contains(nine)==True (1)                                                  -0.090\n",
      "  contains(n)==True (1)                                                     -0.090\n",
      "  contains(fast)==True (1)                                                   0.089\n",
      "  contains(grab)==True (1)                                                   0.088\n",
      "  contains(),)==True (1)                                                     0.088\n",
      "  contains(q)==True (1)                                                     -0.087\n",
      "  contains(passion)==True (1)                                                0.087\n",
      "  contains(pack)==True (1)                                                   0.086\n",
      "  contains(beauty)==True (1)                                                 0.086\n",
      "  contains(get)==True (1)                                                    0.084\n",
      "  contains(rather)==True (1)                                                 0.084\n",
      "  contains(throughout)==True (1)                                            -0.083\n",
      "  contains(whole)==True (1)                                                 -0.083\n",
      "  contains(stick)==True (1)                                                 -0.083\n",
      "  contains(something)==True (1)                                              0.082\n",
      "  contains(spin)==True (1)                                                   0.082\n",
      "  contains(held)==True (1)                                                  -0.082\n",
      "  contains(com)==True (1)                                                   -0.082\n",
      "  contains(kitchen)==True (1)                                               -0.082\n",
      "  contains(tom)==True (1)                                                   -0.081\n",
      "  contains(maybe)==True (1)                                                 -0.080\n",
      "  contains(process)==True (1)                                                0.080\n",
      "  contains(old)==True (1)                                                    0.080\n",
      "  contains(though)==True (1)                                                -0.080\n",
      "  contains(cooking)==True (1)                                               -0.080\n",
      "  contains(much)==True (1)                                                   0.079\n",
      "  contains(crack)==True (1)                                                 -0.079\n",
      "  contains(example)==True (1)                                                0.078\n",
      "  contains(dinner)==True (1)                                                 0.078\n",
      "  contains(decade)==True (1)                                                -0.077\n",
      "  contains(roof)==True (1)                                                  -0.077\n",
      "  contains(paper)==True (1)                                                  0.077\n",
      "  contains(taken)==True (1)                                                  0.076\n",
      "  contains(tomato)==True (1)                                                -0.076\n",
      "  contains(perfectly)==True (1)                                             -0.075\n",
      "  contains(exchange)==True (1)                                               0.075\n",
      "  contains(leave)==True (1)                                                 -0.075\n",
      "  contains(.”)==True (1)                                                    -0.074\n",
      "  contains(sunday)==True (1)                                                -0.073\n",
      "  contains(inspired)==True (1)                                               0.073\n",
      "  contains(developed)==True (1)                                             -0.073\n",
      "  contains(bite)==True (1)                                                  -0.073\n",
      "  contains(deal)==True (1)                                                  -0.073\n",
      "  contains(frank)==True (1)                                                 -0.072\n",
      "  contains(“)==True (1)                                                      0.072\n",
      "  contains(eat)==True (1)                                                    0.072\n",
      "  contains(layer)==True (1)                                                  0.071\n",
      "  contains(exact)==True (1)                                                  0.070\n",
      "  contains(oil)==True (1)                                                    0.070\n",
      "  contains(slightly)==True (1)                                               0.070\n",
      "  contains(25)==True (1)                                                    -0.069\n",
      "  contains(circle)==True (1)                                                 0.068\n",
      "  contains(adam)==True (1)                                                  -0.068\n",
      "  contains(pan)==True (1)                                                    0.067\n",
      "  contains(two)==True (1)                                                   -0.067\n",
      "  contains(kid)==True (1)                                                   -0.067\n",
      "  contains(alike)==True (1)                                                 -0.067\n",
      "  contains(www)==True (1)                                                    0.066\n",
      "  contains(joint)==True (1)                                                 -0.066\n",
      "  contains(.,)==True (1)                                                    -0.066\n",
      "  contains(competition)==True (1)                                           -0.066\n",
      "  contains(set)==True (1)                                                    0.065\n",
      "  contains(butter)==True (1)                                                -0.065\n",
      "  contains(since)==True (1)                                                 -0.065\n",
      "  contains(c)==True (1)                                                      0.065\n",
      "  contains(involved)==True (1)                                               0.065\n",
      "  contains(country)==True (1)                                               -0.065\n",
      "  contains(house)==True (1)                                                 -0.064\n",
      "  contains(—)==True (1)                                                     -0.064\n",
      "  contains(pressure)==True (1)                                               0.064\n",
      "  contains(beer)==True (1)                                                  -0.064\n",
      "  contains(hot)==True (1)                                                   -0.062\n",
      "  contains(plain)==True (1)                                                 -0.062\n",
      "  contains(delicious)==True (1)                                             -0.062\n",
      "  contains(allow)==True (1)                                                  0.061\n",
      "  contains(line)==True (1)                                                  -0.061\n",
      "  contains(ray)==True (1)                                                   -0.061\n",
      "  contains(enough)==True (1)                                                 0.061\n",
      "  contains(child)==True (1)                                                  0.060\n",
      "  contains(father)==True (1)                                                -0.060\n",
      "  contains(lunch)==True (1)                                                  0.059\n",
      "  contains(bring)==True (1)                                                  0.059\n",
      "  contains(12)==True (1)                                                     0.059\n",
      "  contains(washington)==True (1)                                            -0.059\n",
      "  contains(fully)==True (1)                                                 -0.058\n",
      "  contains(east)==True (1)                                                   0.058\n",
      "  contains(juice)==True (1)                                                  0.058\n",
      "  contains(family)==True (1)                                                -0.058\n",
      "  contains(empty)==True (1)                                                  0.057\n",
      "  contains(honey)==True (1)                                                  0.056\n",
      "  contains(adult)==True (1)                                                 -0.056\n",
      "  contains(perfect)==True (1)                                                0.056\n",
      "  contains(element)==True (1)                                               -0.055\n",
      "  contains(silver)==True (1)                                                 0.055\n",
      "  contains(spice)==True (1)                                                 -0.055\n",
      "  contains(golden)==True (1)                                                 0.055\n",
      "  contains(hill)==True (1)                                                  -0.054\n",
      "  contains(french)==True (1)                                                -0.054\n",
      "  contains(secret)==True (1)                                                -0.053\n",
      "  contains(big)==True (1)                                                    0.053\n",
      "  contains(chow)==True (1)                                                   0.053\n",
      "  contains(cast)==True (1)                                                   0.052\n",
      "  contains(snack)==True (1)                                                  0.052\n",
      "  contains(served)==True (1)                                                -0.052\n",
      "  contains(anything)==True (1)                                              -0.051\n",
      "  contains(term)==True (1)                                                  -0.051\n",
      "  contains(planning)==True (1)                                              -0.051\n",
      "  contains(paid)==True (1)                                                   0.050\n",
      "  contains(blend)==True (1)                                                 -0.050\n",
      "  contains(rose)==True (1)                                                  -0.050\n",
      "  contains(pleasure)==True (1)                                              -0.050\n",
      "  contains(accessible)==True (1)                                            -0.050\n",
      "  contains(popular)==True (1)                                               -0.049\n",
      "  contains(combine)==True (1)                                                0.049\n",
      "  contains(getting)==True (1)                                                0.049\n",
      "  contains(spend)==True (1)                                                 -0.049\n",
      "  contains(basically)==True (1)                                             -0.048\n",
      "  contains(operation)==True (1)                                              0.048\n",
      "  contains(stool)==True (1)                                                 -0.048\n",
      "  contains(becky)==True (1)                                                  0.048\n",
      "  contains(actually)==True (1)                                               0.048\n",
      "  contains(another)==True (1)                                                0.048\n",
      "  contains(bonnie)==True (1)                                                 0.047\n",
      "  contains(deeply)==True (1)                                                 0.047\n",
      "  contains(flavor)==True (1)                                                -0.047\n",
      "  contains(american)==True (1)                                              -0.046\n",
      "  contains(ever)==True (1)                                                   0.046\n",
      "  contains(certain)==True (1)                                                0.046\n",
      "  contains(white)==True (1)                                                 -0.045\n",
      "  contains(stuffed)==True (1)                                               -0.045\n",
      "  contains(bolster)==True (1)                                                0.045\n",
      "  contains(time)==True (1)                                                  -0.045\n",
      "  contains(moist)==True (1)                                                  0.045\n",
      "  contains(common)==True (1)                                                 0.044\n",
      "  contains(distributed)==True (1)                                           -0.043\n",
      "  contains(navy)==True (1)                                                   0.043\n",
      "  contains(chicken)==True (1)                                                0.043\n",
      "  contains(bag)==True (1)                                                   -0.042\n",
      "  contains(commitment)==True (1)                                             0.042\n",
      "  contains(protein)==True (1)                                                0.042\n",
      "  contains(list)==True (1)                                                   0.042\n",
      "  contains(95)==True (1)                                                    -0.042\n",
      "  contains(dropping)==True (1)                                              -0.042\n",
      "  contains(named)==True (1)                                                 -0.041\n",
      "  contains(luxury)==True (1)                                                -0.041\n",
      "  contains(offering)==True (1)                                               0.041\n",
      "  contains(bread)==True (1)                                                 -0.041\n",
      "  contains(hold)==True (1)                                                   0.041\n",
      "  contains(fat)==True (1)                                                   -0.041\n",
      "  contains(arrives)==True (1)                                               -0.040\n",
      "  contains(standard)==True (1)                                              -0.040\n",
      "  contains(name)==True (1)                                                  -0.040\n",
      "  contains(taste)==True (1)                                                  0.040\n",
      "  contains(good)==True (1)                                                   0.040\n",
      "  contains(treated)==True (1)                                                0.039\n",
      "  contains(subtle)==True (1)                                                -0.039\n",
      "  contains(dunk)==True (1)                                                   0.038\n",
      "  contains(including)==True (1)                                              0.038\n",
      "  contains(dig)==True (1)                                                   -0.038\n",
      "  contains(frying)==True (1)                                                 0.037\n",
      "  contains(describe)==True (1)                                              -0.037\n",
      "  contains(closely)==True (1)                                                0.037\n",
      "  contains(obsession)==True (1)                                             -0.037\n",
      "  contains(twice)==True (1)                                                  0.037\n",
      "  contains(merit)==True (1)                                                 -0.037\n",
      "  contains(long)==True (1)                                                  -0.036\n",
      "  contains(often)==True (1)                                                  0.036\n",
      "  contains(chili)==True (1)                                                  0.036\n",
      "  contains(newest)==True (1)                                                -0.035\n",
      "  contains(would)==True (1)                                                  0.035\n",
      "  contains(warm)==True (1)                                                   0.035\n",
      "  contains(bird)==True (1)                                                   0.035\n",
      "  contains(agree)==True (1)                                                  0.035\n",
      "  contains(signature)==True (1)                                             -0.034\n",
      "  contains(snap)==True (1)                                                  -0.034\n",
      "  contains(appreciate)==True (1)                                             0.034\n",
      "  contains(breath)==True (1)                                                -0.034\n",
      "  contains(sandwich)==True (1)                                              -0.034\n",
      "  contains(yard)==True (1)                                                   0.034\n",
      "  contains(count)==True (1)                                                 -0.033\n",
      "  contains(word)==True (1)                                                   0.033\n",
      "  contains(glass)==True (1)                                                 -0.033\n",
      "  contains(suit)==True (1)                                                  -0.033\n",
      "  contains(pas)==True (1)                                                   -0.033\n",
      "  contains(excuse)==True (1)                                                -0.033\n",
      "  contains(style)==True (1)                                                  0.032\n",
      "  contains(chef)==True (1)                                                  -0.032\n",
      "  contains(cilantro)==True (1)                                               0.032\n",
      "  contains(distance)==True (1)                                               0.031\n",
      "  contains(brushed)==True (1)                                               -0.031\n",
      "  contains(real)==True (1)                                                   0.031\n",
      "  contains(2002)==True (1)                                                  -0.031\n",
      "  contains(combination)==True (1)                                           -0.030\n",
      "  contains(founded)==True (1)                                               -0.030\n",
      "  contains(proper)==True (1)                                                -0.030\n",
      "  contains(form)==True (1)                                                  -0.029\n",
      "  contains(call)==True (1)                                                  -0.029\n",
      "  contains(sell)==True (1)                                                   0.029\n",
      "  contains(virtue)==True (1)                                                 0.029\n",
      "  contains(stacked)==True (1)                                                0.029\n",
      "  contains(better)==True (1)                                                -0.028\n",
      "  contains(coated)==True (1)                                                 0.028\n",
      "  contains(soul)==True (1)                                                   0.028\n",
      "  contains(teeth)==True (1)                                                  0.028\n",
      "  contains(tempura)==True (1)                                                0.028\n",
      "  contains(back)==True (1)                                                   0.028\n",
      "  contains(matter)==True (1)                                                -0.028\n",
      "  contains(one)==True (1)                                                   -0.027\n",
      "  contains(marriage)==True (1)                                              -0.027\n",
      "  contains(credit)==True (1)                                                -0.027\n",
      "  contains(deep)==True (1)                                                  -0.027\n",
      "  contains(potato)==True (1)                                                 0.027\n",
      "  contains(atop)==True (1)                                                  -0.026\n",
      "  contains(bar)==True (1)                                                   -0.026\n",
      "  contains(keep)==True (1)                                                   0.025\n",
      "  contains(appetite)==True (1)                                              -0.025\n",
      "  contains(richard)==True (1)                                               -0.025\n",
      "  contains(e)==True (1)                                                     -0.025\n",
      "  contains(ultra)==True (1)                                                 -0.025\n",
      "  contains(person)==True (1)                                                -0.025\n",
      "  contains(everywhere)==True (1)                                             0.025\n",
      "  contains(steer)==True (1)                                                 -0.024\n",
      "  contains(knife)==True (1)                                                 -0.024\n",
      "  contains(street)==True (1)                                                 0.024\n",
      "  contains(seasoning)==True (1)                                              0.024\n",
      "  contains(bone)==True (1)                                                  -0.023\n",
      "  contains(ranch)==True (1)                                                  0.023\n",
      "  contains(shine)==True (1)                                                  0.023\n",
      "  contains(taylor)==True (1)                                                -0.023\n",
      "  contains(mac)==True (1)                                                   -0.023\n",
      "  contains(store)==True (1)                                                 -0.022\n",
      "  contains(consistency)==True (1)                                           -0.022\n",
      "  contains(14)==True (1)                                                    -0.022\n",
      "  contains(24)==True (1)                                                    -0.022\n",
      "  contains(sea)==True (1)                                                    0.022\n",
      "  contains(salt)==True (1)                                                  -0.022\n",
      "  contains(buy)==True (1)                                                    0.022\n",
      "  contains(ounce)==True (1)                                                 -0.022\n",
      "  contains(($)==True (1)                                                     0.022\n",
      "  contains(frequently)==True (1)                                             0.022\n",
      "  contains(basket)==True (1)                                                -0.021\n",
      "  contains(steak)==True (1)                                                  0.021\n",
      "  contains(coating)==True (1)                                                0.021\n",
      "  contains(florida)==True (1)                                               -0.021\n",
      "  contains(eric)==True (1)                                                   0.021\n",
      "  contains(preparation)==True (1)                                           -0.021\n",
      "  contains(fan)==True (1)                                                   -0.020\n",
      "  contains(crushed)==True (1)                                                0.020\n",
      "  contains(surprising)==True (1)                                             0.020\n",
      "  contains(version)==True (1)                                                0.019\n",
      "  contains(bath)==True (1)                                                  -0.019\n",
      "  contains(open)==True (1)                                                   0.019\n",
      "  contains(reveal)==True (1)                                                 0.019\n",
      "  contains(technically)==True (1)                                            0.019\n",
      "  contains(deviate)==True (1)                                                0.019\n",
      "  contains(skin)==True (1)                                                   0.019\n",
      "  contains(resist)==True (1)                                                -0.019\n",
      "  contains(high)==True (1)                                                   0.019\n",
      "  contains(pennsylvania)==True (1)                                           0.019\n",
      "  contains(rendered)==True (1)                                               0.019\n",
      "  contains(convenience)==True (1)                                           -0.018\n",
      "  contains(.’)==True (1)                                                    -0.018\n",
      "  contains(got)==True (1)                                                    0.018\n",
      "  contains(happy)==True (1)                                                 -0.018\n",
      "  contains(j)==True (1)                                                     -0.018\n",
      "  contains(columbia)==True (1)                                               0.018\n",
      "  contains(northeast)==True (1)                                              0.018\n",
      "  contains(variety)==True (1)                                                0.018\n",
      "  contains(primary)==True (1)                                                0.017\n",
      "  contains(mixture)==True (1)                                               -0.017\n",
      "  contains(modification)==True (1)                                          -0.017\n",
      "  contains(jersey)==True (1)                                                -0.017\n",
      "  contains(however)==True (1)                                                0.017\n",
      "  contains(dose)==True (1)                                                   0.017\n",
      "  contains(half)==True (1)                                                  -0.017\n",
      "  contains(fry)==True (1)                                                   -0.016\n",
      "  contains(japan)==True (1)                                                 -0.016\n",
      "  contains(snake)==True (1)                                                 -0.016\n",
      "  contains(true)==True (1)                                                   0.016\n",
      "  contains(onion)==True (1)                                                  0.015\n",
      "  contains(garlic)==True (1)                                                 0.015\n",
      "  contains(sold)==True (1)                                                  -0.015\n",
      "  contains(pepper)==True (1)                                                 0.015\n",
      "  contains(turned)==True (1)                                                 0.014\n",
      "  contains(japanese)==True (1)                                              -0.014\n",
      "  contains(follows)==True (1)                                               -0.014\n",
      "  contains(dish)==True (1)                                                   0.014\n",
      "  contains(punishment)==True (1)                                            -0.014\n",
      "  contains(dignity)==True (1)                                                0.014\n",
      "  contains(new)==True (1)                                                   -0.013\n",
      "  contains(serving)==True (1)                                               -0.013\n",
      "  contains(iron)==True (1)                                                  -0.013\n",
      "  contains(formula)==True (1)                                                0.013\n",
      "  contains(accent)==True (1)                                                 0.013\n",
      "  contains(presentation)==True (1)                                          -0.013\n",
      "  contains(inevitably)==True (1)                                            -0.013\n",
      "  contains(toward)==True (1)                                                 0.012\n",
      "  contains(fly)==True (1)                                                    0.012\n",
      "  contains(remains)==True (1)                                               -0.012\n",
      "  contains(flash)==True (1)                                                  0.012\n",
      "  contains(sole)==True (1)                                                   0.012\n",
      "  contains(side)==True (1)                                                  -0.012\n",
      "  contains(sink)==True (1)                                                  -0.011\n",
      "  contains(modern)==True (1)                                                -0.011\n",
      "  contains(generous)==True (1)                                              -0.011\n",
      "  contains(meat)==True (1)                                                  -0.011\n",
      "  contains(emerge)==True (1)                                                -0.010\n",
      "  contains(dad)==True (1)                                                   -0.010\n",
      "  contains(discover)==True (1)                                              -0.010\n",
      "  contains(hartman)==True (1)                                                0.010\n",
      "  contains(chain)==True (1)                                                 -0.010\n",
      "  contains(unless)==True (1)                                                -0.010\n",
      "  contains(messy)==True (1)                                                 -0.010\n",
      "  contains(tender)==True (1)                                                -0.010\n",
      "  contains(dip)==True (1)                                                   -0.010\n",
      "  contains(spoon)==True (1)                                                  0.009\n",
      "  contains(home)==True (1)                                                   0.009\n",
      "  contains(nashville)==True (1)                                             -0.009\n",
      "  contains(pain)==True (1)                                                  -0.009\n",
      "  contains(flesh)==True (1)                                                  0.009\n",
      "  contains(forgotten)==True (1)                                             -0.009\n",
      "  contains(estate)==True (1)                                                 0.009\n",
      "  contains(filling)==True (1)                                                0.008\n",
      "  contains(longer)==True (1)                                                -0.008\n",
      "  contains(favorite)==True (1)                                               0.008\n",
      "  contains(pool)==True (1)                                                   0.008\n",
      "  contains(spring)==True (1)                                                -0.008\n",
      "  contains(se)==True (1)                                                     0.008\n",
      "  contains(lends)==True (1)                                                  0.008\n",
      "  contains(excursion)==True (1)                                              0.007\n",
      "  contains(end)==True (1)                                                    0.007\n",
      "  contains(within)==True (1)                                                 0.007\n",
      "  contains(crunch)==True (1)                                                 0.007\n",
      "  contains(shattering)==True (1)                                            -0.007\n",
      "  contains(waste)==True (1)                                                 -0.007\n",
      "  contains(fairfax)==True (1)                                                0.006\n",
      "  contains(corn)==True (1)                                                  -0.006\n",
      "  contains(heavy)==True (1)                                                 -0.006\n",
      "  contains(favor)==True (1)                                                 -0.006\n",
      "  contains(ordering)==True (1)                                              -0.006\n",
      "  contains(whether)==True (1)                                                0.006\n",
      "  contains(traditional)==True (1)                                           -0.006\n",
      "  contains(georgia)==True (1)                                               -0.005\n",
      "  contains(memorable)==True (1)                                             -0.005\n",
      "  contains(even)==True (1)                                                   0.005\n",
      "  contains(thigh)==True (1)                                                 -0.005\n",
      "  contains(area)==True (1)                                                  -0.005\n",
      "  contains(plate)==True (1)                                                  0.005\n",
      "  contains(upscale)==True (1)                                                0.005\n",
      "  contains(popcorn)==True (1)                                                0.005\n",
      "  contains(yield)==True (1)                                                  0.005\n",
      "  contains(tongue)==True (1)                                                 0.005\n",
      "  contains(generously)==True (1)                                            -0.004\n",
      "  contains(880)==True (1)                                                    0.004\n",
      "  contains(unadorned)==True (1)                                              0.004\n",
      "  contains(starch)==True (1)                                                 0.004\n",
      "  contains(rested)==True (1)                                                 0.003\n",
      "  contains(ear)==True (1)                                                   -0.003\n",
      "  contains(commonly)==True (1)                                               0.003\n",
      "  contains(without)==True (1)                                                0.003\n",
      "  contains(meal)==True (1)                                                   0.003\n",
      "  contains(essentially)==True (1)                                           -0.002\n",
      "  contains(consciousness)==True (1)                                          0.002\n",
      "  contains(fritz)==True (1)                                                 -0.002\n",
      "  contains(available)==True (1)                                             -0.002\n",
      "  contains(crystal)==True (1)                                                0.002\n",
      "  contains(consistently)==True (1)                                          -0.002\n",
      "  contains(wheat)==True (1)                                                 -0.001\n",
      "  contains(respect)==True (1)                                                0.001\n",
      "  contains(bun)==True (1)                                                   -0.001\n",
      "  contains(occasional)==True (1)                                            -0.001\n",
      "  contains(dedicated)==True (1)                                             -0.001\n",
      "  contains(satisfying)==True (1)                                            -0.000\n",
      "  contains(latching)==True (1)                                               0.000\n",
      "  contains(vernon)==True (1)                                                 0.000\n",
      "  contains(karaoke)==True (1)                                                0.000\n",
      "  ---------------------------------------------------------------------------------\n",
      "  TOTAL:                                            40.166   9.835   9.084   6.412\n",
      "  PROBS:                                             1.000   0.000   0.000   0.000\n"
     ]
    }
   ],
   "source": [
    "from operator import itemgetter\n",
    "\n",
    "def classify(text, explain=False):\n",
    "    \n",
    "    classifier = None\n",
    "    with open('classifier.pickle', 'rb') as f:\n",
    "        classifier = pickle.load(f)\n",
    "    \n",
    "    document = nltk.wordpunct_tokenize(text)\n",
    "    features = document_features(document)\n",
    "    \n",
    "    pd = classifier.prob_classify(features)\n",
    "    for result in sorted([(s,pd.prob(s)) for s in pd.samples()], key=itemgetter(1), reverse=True):\n",
    "        print \"%s: %0.4f\" % result\n",
    "\n",
    "    print\n",
    "    if explain:\n",
    "        classifier.explain(features)\n",
    "\n",
    "classify(get_text(\"nrRB0.html\"), True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  Feature                                           design   books data_sc    tech\n",
      "  --------------------------------------------------------------------------------\n",
      "  contains(’)==True (1)                              1.444\n",
      "  contains(2)==True (1)                              0.709\n",
      "  contains(x)==True (1)                              0.607\n",
      "  contains(4)==True (1)                              0.513\n",
      "  contains(h)==True (1)                              0.475\n",
      "  contains(“)==True (1)                              0.368\n",
      "  label is 'design' (1)                              0.362\n",
      "  contains(r)==True (1)                              0.331\n",
      "  contains(0)==True (1)                             -0.296\n",
      "  contains(7)==True (1)                             -0.290\n",
      "  contains(8)==True (1)                             -0.266\n",
      "  contains(v)==True (1)                             -0.261\n",
      "  contains(q)==True (1)                              0.255\n",
      "  contains(9)==True (1)                             -0.237\n",
      "  contains(—)==True (1)                             -0.228\n",
      "  contains(l)==True (1)                              0.210\n",
      "  contains(1)==True (1)                              0.182\n",
      "  contains(f)==True (1)                             -0.161\n",
      "  contains(c)==True (1)                             -0.115\n",
      "  contains(”)==True (1)                             -0.106\n",
      "  contains(‘)==True (1)                             -0.093\n",
      "  contains(w)==True (1)                              0.092\n",
      "  contains(e)==True (1)                             -0.091\n",
      "  contains(5)==True (1)                              0.086\n",
      "  contains(z)==True (1)                             -0.077\n",
      "  contains(n)==True (1)                             -0.066\n",
      "  contains(b)==True (1)                              0.050\n",
      "  contains(j)==True (1)                              0.044\n",
      "  contains(3)==True (1)                              0.038\n",
      "  contains(p)==True (1)                             -0.031\n",
      "  contains(k)==True (1)                              0.026\n",
      "  contains(6)==True (1)                             -0.004\n",
      "  contains(u)==True (1)                              0.003\n",
      "  contains(e)==True (1)                                      1.766\n",
      "  contains(“)==True (1)                                      1.441\n",
      "  contains(‘)==True (1)                                      1.095\n",
      "  contains(9)==True (1)                                      0.865\n",
      "  contains(’)==True (1)                                      0.745\n",
      "  contains(”)==True (1)                                      0.577\n",
      "  contains(u)==True (1)                                     -0.520\n",
      "  label is 'books' (1)                                      -0.416\n",
      "  contains(5)==True (1)                                     -0.388\n",
      "  contains(3)==True (1)                                     -0.358\n",
      "  contains(6)==True (1)                                     -0.263\n",
      "  contains(2)==True (1)                                     -0.236\n",
      "  contains(8)==True (1)                                     -0.215\n",
      "  contains(1)==True (1)                                     -0.212\n",
      "  contains(x)==True (1)                                     -0.190\n",
      "  contains(—)==True (1)                                     -0.177\n",
      "  contains(v)==True (1)                                     -0.176\n",
      "  contains(4)==True (1)                                     -0.157\n",
      "  contains(r)==True (1)                                     -0.150\n",
      "  contains(b)==True (1)                                     -0.096\n",
      "  contains(z)==True (1)                                     -0.052\n",
      "  contains(n)==True (1)                                     -0.051\n",
      "  contains(j)==True (1)                                     -0.050\n",
      "  contains(7)==True (1)                                     -0.048\n",
      "  contains(k)==True (1)                                      0.042\n",
      "  contains(g)==True (1)                                     -0.036\n",
      "  contains(c)==True (1)                                     -0.034\n",
      "  contains(f)==True (1)                                      0.017\n",
      "  contains(l)==True (1)                                      0.014\n",
      "  contains(p)==True (1)                                      0.002\n",
      "  contains(r)==True (1)                                              1.170\n",
      "  contains(—)==True (1)                                             -0.854\n",
      "  label is 'data_science' (1)                                        0.606\n",
      "  contains(”)==True (1)                                              0.393\n",
      "  contains(4)==True (1)                                             -0.331\n",
      "  contains(v)==True (1)                                              0.287\n",
      "  contains(7)==True (1)                                             -0.285\n",
      "  contains(g)==True (1)                                              0.264\n",
      "  contains(’)==True (1)                                              0.257\n",
      "  contains(6)==True (1)                                              0.225\n",
      "  contains(n)==True (1)                                              0.223\n",
      "  contains(3)==True (1)                                             -0.221\n",
      "  contains(“)==True (1)                                             -0.194\n",
      "  contains(8)==True (1)                                             -0.185\n",
      "  contains(j)==True (1)                                              0.166\n",
      "  contains(1)==True (1)                                              0.138\n",
      "  contains(x)==True (1)                                              0.093\n",
      "  contains(l)==True (1)                                             -0.091\n",
      "  contains(q)==True (1)                                             -0.077\n",
      "  contains(h)==True (1)                                             -0.075\n",
      "  contains(e)==True (1)                                              0.073\n",
      "  contains(5)==True (1)                                             -0.067\n",
      "  contains(f)==True (1)                                             -0.066\n",
      "  contains(w)==True (1)                                              0.057\n",
      "  contains(9)==True (1)                                              0.055\n",
      "  contains(b)==True (1)                                             -0.052\n",
      "  contains(u)==True (1)                                              0.040\n",
      "  contains(z)==True (1)                                              0.039\n",
      "  contains(p)==True (1)                                              0.032\n",
      "  contains(k)==True (1)                                             -0.032\n",
      "  contains(‘)==True (1)                                             -0.032\n",
      "  contains(0)==True (1)                                              0.028\n",
      "  contains(c)==True (1)                                              0.026\n",
      "  contains(2)==True (1)                                              0.004\n",
      "  contains(8)==True (1)                                                      1.563\n",
      "  contains(—)==True (1)                                                      1.497\n",
      "  contains(2)==True (1)                                                     -0.904\n",
      "  contains(r)==True (1)                                                     -0.437\n",
      "  contains(f)==True (1)                                                      0.431\n",
      "  contains(4)==True (1)                                                     -0.414\n",
      "  contains(‘)==True (1)                                                     -0.409\n",
      "  contains(u)==True (1)                                                      0.389\n",
      "  contains(5)==True (1)                                                     -0.344\n",
      "  contains(“)==True (1)                                                     -0.323\n",
      "  contains(v)==True (1)                                                      0.295\n",
      "  label is 'tech' (1)                                                        0.278\n",
      "  contains(0)==True (1)                                                      0.272\n",
      "  contains(9)==True (1)                                                     -0.243\n",
      "  contains(7)==True (1)                                                      0.238\n",
      "  contains(6)==True (1)                                                      0.213\n",
      "  contains(k)==True (1)                                                     -0.141\n",
      "  contains(1)==True (1)                                                     -0.134\n",
      "  contains(3)==True (1)                                                     -0.120\n",
      "  contains(p)==True (1)                                                     -0.118\n",
      "  contains(b)==True (1)                                                      0.102\n",
      "  contains(n)==True (1)                                                     -0.101\n",
      "  contains(j)==True (1)                                                     -0.091\n",
      "  contains(q)==True (1)                                                      0.067\n",
      "  contains(”)==True (1)                                                     -0.058\n",
      "  contains(h)==True (1)                                                     -0.056\n",
      "  contains(z)==True (1)                                                      0.052\n",
      "  contains(c)==True (1)                                                     -0.044\n",
      "  contains(l)==True (1)                                                     -0.040\n",
      "  contains(’)==True (1)                                                      0.037\n",
      "  contains(g)==True (1)                                                     -0.036\n",
      "  contains(e)==True (1)                                                     -0.035\n",
      "  contains(w)==True (1)                                                     -0.026\n",
      "  contains(x)==True (1)                                                     -0.023\n",
      "  ---------------------------------------------------------------------------------\n",
      "  TOTAL:                                             3.473   2.739   1.617   1.338\n",
      "  PROBS:                                             0.378   0.227   0.104   0.086\n"
     ]
    }
   ],
   "source": [
    "classifier.explain(document_features(get_text(\"nrRB0.html\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The classifier did well - it trained in 2 minutes or so an dit got an initial accuracy of about 83% - a pretty good start!\n",
    "\n",
    "### Parsing with Stanford Parser and NLTK\n",
    "\n",
    "NLTK parsing is notoriously bad - because it's pedagogical. However, you can use Stanford. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "from nltk.tag.stanford import NERTagger\n",
    "from nltk.parse.stanford import StanfordParser\n",
    "\n",
    "## NER JAR and Models\n",
    "STANFORD_NER_MODEL = os.path.expanduser(\"~/Development/stanford-ner-2014-01-04/classifiers/english.all.3class.distsim.crf.ser.gz\")\n",
    "STANFORD_NER_JAR   = os.path.expanduser(\"~/Development/stanford-ner-2014-01-04/stanford-ner-2014-01-04.jar\")\n",
    "\n",
    "## Parser JAR and Models\n",
    "STANFORD_PARSER_MODELS = os.path.expanduser(\"~/Development/stanford-parser-full-2014-10-31/stanford-parser-3.5.0-models.jar\")\n",
    "STANFORD_PARSER_JAR    = os.path.expanduser(\"~/Development/stanford-parser-full-2014-10-31/stanford-parser.jar\")\n",
    "\n",
    "def create_tagger(model=None, jar=None, encoding='ASCII'):\n",
    "    model = model or STANFORD_NER_MODEL\n",
    "    jar   = jar or STANFORD_NER_JAR\n",
    "\n",
    "    return NERTagger(model, jar, encoding)\n",
    "\n",
    "def create_parser(models=None, jar=None, **kwargs):\n",
    "    models = models or STANFORD_PARSER_MODELS\n",
    "    jar   = jar or STANFORD_PARSER_JAR\n",
    "\n",
    "    return StanfordParser(jar, models, **kwargs)\n",
    "\n",
    "class NER(object):\n",
    "\n",
    "    tagger = None\n",
    "\n",
    "    @classmethod\n",
    "    def initialize_tagger(klass, model=None, jar=None, encoding='ASCII'):\n",
    "        klass.tagger = create_tagger(model, jar, encoding)\n",
    "\n",
    "    @classmethod\n",
    "    def tag(klass, sent):\n",
    "        if klass.tagger is None:\n",
    "            klass.initialize_tagger()\n",
    "\n",
    "        sent = nltk.word_tokenize(sent)\n",
    "        return klass.tagger.tag(sent)\n",
    "\n",
    "class Parser(object):\n",
    "\n",
    "    parser = None\n",
    "\n",
    "    @classmethod\n",
    "    def initialize_parser(klass, models=None, jar=None, **kwargs):\n",
    "        klass.parser = create_parser(models, jar, **kwargs)\n",
    "\n",
    "    @classmethod\n",
    "    def parse(klass, sent):\n",
    "        if klass.parser is  None:\n",
    "            klass.initialize_parser()\n",
    "\n",
    "        return klass.parser.raw_parse(sent)\n",
    "\n",
    "def tag(sent):\n",
    "    return NER.tag(sent)\n",
    "\n",
    "def parse(sent):\n",
    "    return Parser.parse(sent)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(u'The', u'O'),\n",
       " (u'man', u'O'),\n",
       " (u'hit', u'O'),\n",
       " (u'the', u'O'),\n",
       " (u'building', u'O'),\n",
       " (u'with', u'O'),\n",
       " (u'the', u'O'),\n",
       " (u'bat', u'O'),\n",
       " (u'.', u'O')]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tag(\"The man hit the building with the bat.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(ROOT\n",
      "  (S\n",
      "    (NP (DT The) (NN man))\n",
      "    (VP\n",
      "      (VBD hit)\n",
      "      (NP (DT the) (NN building))\n",
      "      (PP (IN with) (NP (DT the) (NN bat))))\n",
      "    (. .)))\n"
     ]
    }
   ],
   "source": [
    "for p in parse(\"The man hit the building with the bat.\"):\n",
    "    print p\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TextBlob\n",
    "\n",
    "A lightweight wrapper around nltk that provides a simple \"Blob\" interface for working with text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Sentence(\"A crisp and juicy bucket list of D.C.’s best fried chicken - The Washington Post\n",
      "\n",
      " It’s lowbrow.\"), Sentence(\"It’s messy.\"), Sentence(\"It could never be accused of being healthful.\"), Sentence(\"But we’d never let those formalities get between us and an order of crispy, crackly, delicious fried chicken.\"), Sentence(\"Whether it comes in a bucket or on a bun, or you eat it with your fingers or chopsticks, there’s a surprising variety to the Washington area’s fried chicken offerings.\"), Sentence(\"Here are some of the most irresistible.\"), Sentence(\"‘Rotissi-fried’ chicken at the Partisan\n",
      "\n",
      "Forget the cronut.\"), Sentence(\"Our newest favorite food chimera is the “rotissi-fried” chicken at the Partisan.\"), Sentence(\"Credit goes to chef Nate Anda, who dreamed up the dish: After a 12-hour brine, the chicken is rotisseried for two hours and then fried for two and a half minutes.\"), Sentence(\"Why both?\"), Sentence(\"“Everything is better once it’s fried in beef fat,” Anda said.\"), Sentence(\"We have to agree.\"), Sentence(\"Whether white or dark, the meat is succulent throughout.\"), Sentence(\"The batter-free frying leaves the simply seasoned skin rendered perfectly crisp, golden and translucent — cracklings, essentially.\"), Sentence(\"The sound of it shattering under the knife was music to our ears.\"), Sentence(\"And as if the lily needed further gilding, the chicken comes with a generous pour of honey hot sauce.\"), Sentence(\"The sauce is hard to resist, but try to reserve a few bites of unadorned chicken so you can fully appreciate this happy marriage of classic preparations.\"), Sentence(\"The Partisan, 709 D St. NW.\"), Sentence(\"202-524-5322. www.thepartisandc.com.\"), Sentence(\"— Becky Krystal\n",
      "\n",
      " [In a love/hate relationship with Chick-fil-A?\"), Sentence(\"Here are some alternatives] \n",
      "\n",
      "Traditional fried chicken at Family Meal\n",
      "\n",
      "When Bryan Voltaggio started planning the menu for Family Meal, his modern, upscale spin on a diner, his thoughts turned to home and the carryout meal he most enjoyed as a kid: fried chicken.\"), Sentence(\"“It was one of our favorite things,” he says.\"), Sentence(\"“It just seems like a family dinner.” And if he was creating a restaurant called Family Meal, fried chicken “had to be an important part of it.” But Voltaggio wanted to do it right, and set about testing cooking methods, brines, breadings and fryers.\"), Sentence(\"That long process paid off with a home run of a fried chicken dish that’s become the most popular item on Family Meal’s menu.\"), Sentence(\"The whole chickens spend 12 hours in a brine of pickle juice and roasted poultry stock before getting dredged, rested and dredged again in a mixture of flour, cornmeal and corn starch.\"), Sentence(\"After a dip in the top-of-the-line pressure fryer, the thighs, legs and breasts emerge with a crisp, salty skin that cracks open to reveal wonderfully warm and moist flesh.\"), Sentence(\"You don’t even need to dunk it in the house-made hot sauce that accompanies the dish, but really, who can resist?\"), Sentence(\"Family Meal Ashburn, 20462 Exchange St. 703-726-9800; Frederick, 880 N. East St. 301-378-2895; Baltimore, 621 E. Pratt St. 410-601-3242. www.voltfamilymeal.com.\"), Sentence(\"— John Taylor\n",
      "\n",
      "  \n",
      "\n",
      " [40 Eats: D.C.’s most essential dishes of 2015] \n",
      "\n",
      "\n",
      "\n",
      "Japanese fried chicken at Izakaya Seki\n",
      "\n",
      "Although it’s commonly served in Japan at karaoke bars, convenience stores and on street carts, karaage chicken — like most of the country’s food — is held to an extremely high standard.\"), Sentence(\"“It’s taken it to the nth degree of obsession and detail,” says Cizuka Seki, who, with her father Hiroshi, owns Izakaya Seki on V Street NW.\"), Sentence(\"“Kara age” is used to describe the method for deep-frying bite-size pieces of fish and, more commonly, chicken.\"), Sentence(\"Though there are subtle variations on the ubiquitous dish, most recipes call for chicken thighs marinated in soy sauce, coated in flour or corn starch and deep-fried in oil.\"), Sentence(\"Izakaya Seki’s version sticks closely to the formula.\"), Sentence(\"Probably.\"), Sentence(\"“I’m not even quite sure what my dad puts into it, because we don’t have recipes,” Seki says, though she’s certain wheat flour is involved.\"), Sentence(\"The result is a thin, tender coating that’s slightly softer than tempura.\"), Sentence(\"The accompanying ponzu sauce lends a tartness to the nubs.\"), Sentence(\"Izakaya Seki, 1117 V St. NW.\"), Sentence(\"202-588-5841. www.sekidc.com.\"), Sentence(\"— Holley Simmons\n",
      "\n",
      "Korean fried chicken at BonChon\n",
      "\n",
      "Don’t waste your kimchi-stinking breath asking for more sauce at BonChon.\"), Sentence(\"The South Korean fried chicken chain, founded in 2002, is so dedicated to consistency that it doesn’t allow for any modifications.\"), Sentence(\"And why would you want to change anything, really?\"), Sentence(\"The made-to-order wings, drumsticks and strips are fried twice, resulting in a paper-thin crust that yields the same satisfying crack as shattering crème brulee with a spoon.\"), Sentence(\"Founder Jinduk Seh spent two years perfecting his secret sauces, which come in three flavors — soy garlic, hot and a blend of the two — and are brushed on by hand post-fry, piece by piece.\"), Sentence(\"True to BonChon’s commitment to uniformity, sauces are made exclusively in South Korea and distributed to all 140-plus BonChon locations, which means the wings you’re chewing on in Arlington are slathered with the same exact stuff as those in the Philippines.\"), Sentence(\"Joints like these are so common throughout Korea they’re called “chimeks,” which is a hybrid term that combines “chicken” with the Korean word for beer.\"), Sentence(\"Washington should be happy to have 10 BonChons within driving distance, plus a brand new Metro-accessible location near the Navy Yard.\"), Sentence(\"BonChon, 1015 Half St.\"), Sentence(\"SE and nine other locations in Maryland and Virginia.\"), Sentence(\"www.bonchon.com.\"), Sentence(\"— Holley Simmons\n",
      "\n",
      "Maryland fried chicken at Crisfield Seafood and Hank’s Oyster Bar\n",
      "\n",
      "There’s not much agreement on what constitutes Maryland fried chicken.\"), Sentence(\"Some say it’s just a fresh Maryland chicken that’s pan-fried; others say it should be topped with white gravy, almost like a chicken-fried steak.\"), Sentence(\"The pan-fried chicken platter at  Crisfield Seafood is a perfect example of the former style.\"), Sentence(\"Half of a chicken is dredged in flour, dusted with salt and pepper, and fried in a cast-iron pan.\"), Sentence(\"This preparation lends a snap and crunch to the exterior, and while the meat falls off the bone, the well-seasoned breading holds on.\"), Sentence(\"(The chicken is available only Friday through Sunday, and frequently sells out.)\"), Sentence(\"The Chesapeake fried chicken at Hank’s Oyster Bar in Dupont Circle and Capitol Hill is plumper than Crisfield’s version and seasoned with Old Bay, black pepper and cayenne, but the breading is softer and less crispy.\"), Sentence(\"It’s brined for 24 hours and deep-fried, rather than pan-fried, and it’s served only on Sunday.\"), Sentence(\"Crisfield Seafood, 8012 Georgia Ave., Silver Spring.\"), Sentence(\"301-589-1306. www.crisfieldseafood.com.\"), Sentence(\"Hank’s Oyster Bar, 1624 Q St. NW.\"), Sentence(\"202-462-4265; 633 Pennsylvania Ave.\"), Sentence(\"SE.\"), Sentence(\"202-733-1971. www.hanksoysterbar.com.\"), Sentence(\"— Fritz Hahn\n",
      "\n",
      "\n",
      "\n",
      " [Pizza in Washington: An upper-crust tour of every D.C. style ]\n",
      "\n",
      "\n",
      "\n",
      "Fancy fried chicken at Central\n",
      "\n",
      "Self-consciousness may prevent you from ordering fried chicken in a white-tablecloth restaurant.\"), Sentence(\"It feels incongruous — gauche, almost — to dig into picnic fare at the kind of place where you should be ordering risotto or tartare or something that comes with mousse, gelee or foam.\"), Sentence(\"But you have to override that adult voice in the back of your head, because if you don’t, you’ll miss out on Central Michel Richard’s famed fried chicken plate ($24 at lunch, $25 for dinner), which remains as good as ever.\"), Sentence(\"Though it’s no longer sold by the bucket to go, Michel Richard’s KFC-inspired crispy breast and thigh come stacked atop a pool of the butteriest mashed potatoes you’ll ever taste.\"), Sentence(\"It’s such a dignified presentation that this most American of dishes almost could pass as (gasp!)\"), Sentence(\"French.\"), Sentence(\"Self-consciousness should, however, steer you toward using a knife and fork — not your fingers — to eat the chicken.\"), Sentence(\"It is, after all, that kind of a place.\"), Sentence(\"Central Michel Richard, 1001 Pennsylvania Ave. NW.\"), Sentence(\"202-626-0015. www.centralmichelrichard.com .\"), Sentence(\"— Maura Judkis\n",
      "\n",
      "\n",
      "\n",
      "Nashville hot chicken at Reserve 2216\n",
      "\n",
      "If you believe the lore, Nashville hot chicken was basically a crime of passion, created as a blistering rebuke to a no-account Romeo who couldn’t keep his hands off other women.\"), Sentence(\"Alas, this wolf was also a chili head who found pleasure, not pain, in this dish of revenge served hot.\"), Sentence(\"Decades later, chefs are latching onto this addictive form of punishment.\"), Sentence(\"Aaron Silverman served an ultra-refined version at Rose’s Luxury for months, and now Eric Reid, chef and co-owner of Reserve 2216 in Del Ray, has developed his own take on hot chicken, even if he’s never actually enjoyed it in Nashville.\"), Sentence(\"He marinates an airline cut (boneless breast with the drumette wing attached) in buttermilk and Crystal hot sauce before dredging the chicken in seasoned flour and frying it.\"), Sentence(\"Reid ditches the traditional white bread base in favor of collards and a side of corn bread waffles.\"), Sentence(\"He finishes the dish with a combination of Cajun seasonings and more Crystal hot sauce for a moist, crispy bird that bites back.\"), Sentence(\"But not too hard.\"), Sentence(\"This is Alexandria, after all.\"), Sentence(\"Reserve 2216, 2216 Mount Vernon Ave., Alexandria.\"), Sentence(\"703-549-2889. www.drpreserve.com.\"), Sentence(\"— Tim Carman\n",
      "\n",
      "\n",
      "\n",
      "Fast-food fried chicken at Popeyes\n",
      "\n",
      "The sole virtue of most fast-food operations is consistency.\"), Sentence(\"Whether you bite into a Big Mac in Bethesda or Beijing, the sandwich should taste the same.\"), Sentence(\"The menu at Popeyes follows suit, but it deviates from the brand-name competition in an important respect: The signature at Popeyes could pass for home cooking (well, if your home had a vat of clean, hot oil and a person with a Southern accent tending the meal).\"), Sentence(\"Maybe that accounts for my occasional forays to the chicken fryer after a bum restaurant-review excursion.\"), Sentence(\"No matter where I eat my order, inevitably “spicy,” I know I can count on a coating that smacks of cayenne, paprika and even crushed cornflakes, and chicken that spurts with juice.\"), Sentence(\"The shatter is audible; the golden crumbs fly everywhere, but end up on my tongue.\"), Sentence(\"No one-trick pony, Popeyes has hot and tender buttermilk biscuits that bolster my favorite excuse to snack low on the food chain.\"), Sentence(\"Once, I got home to discover a clerk had forgotten to pack bread in my bag.\"), Sentence(\"I almost cried.\"), Sentence(\"Instead, I consoled myself with another piece of chicken.\"), Sentence(\"Popeyes has locations throughout the D.C. metro area.\"), Sentence(\"www.popeyes.com.\"), Sentence(\"— Tom Sietsema\n",
      "\n",
      "\n",
      "\n",
      "Fried chicken sandwich at DCity Smokehouse\n",
      "\n",
      "The fried chicken sandwich hasn’t been the same since KFC’s Double Down turned a guilty pleasure into an outright farce, using crispy white-meat fillets as both the sandwich’s primary protein and the oily handles by which you eat the monstrosity.\"), Sentence(\"Leave it to Rob Sonderman, pitmaster and co-owner of DCity Smokehouse, to bring dignity back to the bite.\"), Sentence(\"His Den-Den — named for co-creator and pitmaster-in-training Dennis Geddie — begins with boneless thighs marinated in buttermilk, hot sauce and honey.\"), Sentence(\"Sonderman then dredges the meat in seasoned flour before dropping the thighs into the fryer.\"), Sentence(\"Generously stuffed into a grilled hoagie roll with lettuce, tomato and crispy onions, the chicken is finished with two sauces, including a house-made cilantro ranch.\"), Sentence(\"Technically, the Den-Den ($9.25) is one of the few smokeless items on DCity’s menu (unless you count the chipotle peppers in the hot sauce).\"), Sentence(\"No matter.\"), Sentence(\"You won’t care the minute you sink your teeth into that heaping hoagie of spicy thigh meat.\"), Sentence(\"DCity Smokehouse, 8 Florida Ave. NW.\"), Sentence(\"202-733-1919. www.dcitysmokehouse.com.\"), Sentence(\"— Tim Carman\n",
      "\n",
      "\n",
      "\n",
      "Classic D.C. fried chicken at Oohh’s and Aahh’s\n",
      "\n",
      "Hearty is the appetite that can handle Oohh’s and Aahh’s chef-owner Oji Abbott’s boneless fried chicken breast without taking home leftovers.\"), Sentence(\"He buys local — from Hartman Meat Co. in Northeast Washington — and butterflies each 14-ounce portion ($12.95), which results in a lot of real estate for the crispy, well-seasoned coating.\"), Sentence(\"Abbott chalks up the consistently moist meat to proper cooking time and temperature, and to the recipe he learned from his grandmother.\"), Sentence(\"Oohh’s and Aahh’s, 1005 U St. NW.\"), Sentence(\"202-667-7142. www.oohhsnaahhs.com.\"), Sentence(\"— Bonnie S. Benwick\n",
      "\n",
      "\n",
      "\n",
      "Popcorn fried chicken at Pop’s Sea Bar\n",
      "\n",
      "It’s all too easy to chomp through an order of Boardwalk Chicken at the shore-happy Pop’s Sea Bar in Adams Morgan.\"), Sentence(\"The bite-size pieces of dark-meat-only bird are brined for two hours, then treated to a buttermilk bath until they are coated with plain flour and flash-fried to order.\"), Sentence(\"A generous hand with salt and pepper just before serving means the effect of that seasoning builds as you empty the single-serving basket ($8.99).\"), Sentence(\"Ask for two portions of the accompanying Jersey sauce so you’ll have enough of its horseradish-y, kitchen-sink blend for every bite.\"), Sentence(\"Pop’s Sea Bar, 1817 Columbia Rd.\"), Sentence(\"NW.\"), Sentence(\"202-534-3933. www.popsseabar.com.\"), Sentence(\"— Bonnie S. Benwick\n",
      "\n",
      "\n",
      "\n",
      "Fried chicken tenders at GBD\n",
      "\n",
      "Chicken tenders are often relegated to the children’s menu, but the chicken Tendies at GBD are a fine meal for adults and children alike.\"), Sentence(\"Each white-meat tender comes with a dark, crispy outer layer with a heavy dose of salt and spice.\"), Sentence(\"But the best part about chicken tenders is the dipping, and GBD offers nine sauces, including a take on D.C.’s own mumbo sauce, buttermilk ranch, chipotle barbecue and Frankenbutter, which combines Frank’s RedHot sauce with butter.\"), Sentence(\"Ask for the $5.50 Saucetown option to try all nine.\"), Sentence(\"GBD, 1323 Connecticut Ave. NW.\"), Sentence(\"202-524-5210. www.gbdchickendoughnuts.com.\"), Sentence(\"— Margaret Ely\n",
      "\n",
      "\n",
      "\n",
      "Fried chicken skins at Gypsy Soul\n",
      "\n",
      "Fried chicken fans can argue the merits of white meat vs. dark meat, or whether it’s better to chow down on a drumstick or the breast.\"), Sentence(\"But one thing we all can agree on is that the outer layer — the breading and the skin — is the most important element of a memorable piece of fried chicken.\"), Sentence(\"And sometimes you just want to savor the flavor of the skin — deeply spiced, perfect crunch — without filling up on meat or having to deal with bones.\"), Sentence(\"And that is when you grab one of the bar stools at R.J. Cooper’s Gypsy Soul in Fairfax’s Mosaic District.\"), Sentence(\"Cooper’s chicken skins ($9) are twisted slivers and shards, decadently salty and crackling with paprika, cayenne pepper and garlic.\"), Sentence(\"The dish arrives with a house-made “roof top honey-snake oil,” but it’s best to let these beauties shine on their own.\"), Sentence(\"Gypsy Soul, 8296 Glass Alley, Fairfax.\"), Sentence(\"703-992-0933. www.gypsysoul-va.com.\"), Sentence(\"— Fritz Hahn\n",
      "\n",
      " Related items: \n",
      "\n",
      " D.C.’s most essential dishes of 2015 \n",
      "\n",
      " Pizza in Washington: An upper crust tour of every D.C. style \n",
      "\n",
      "\n",
      "\n",
      "\")]\n"
     ]
    }
   ],
   "source": [
    "from textblob import TextBlob\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "text = TextBlob(get_text(\"nrRB0.html\"))\n",
    "\n",
    "print text.sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(u'hot sauce', 6), (u'popeyes', 5), (u'washington', 5), (u'it\\u2019s', 4), (u'gbd', 4), (u'd.c.\\u2019s', 4), (u'st. nw', 4), (u'bonchon', 3), (u'maryland', 3), (u'hank\\u2019s oyster', 3)]\n"
     ]
    }
   ],
   "source": [
    "np = nltk.FreqDist(text.noun_phrases)\n",
    "print np.most_common(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment(polarity=-0.0025676717918097208, subjectivity=0.5856343297507093)\n"
     ]
    }
   ],
   "source": [
    "print text.sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment(polarity=0.4555555555555555, subjectivity=0.8083333333333333)\n"
     ]
    }
   ],
   "source": [
    "review = TextBlob(\"Harrison Ford would be the most amazing, most wonderful, most handsome actor - the greatest that ever lived, if only he didn't have that silly earing.\")\n",
    "print review.sentiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Language Detection using TextBlob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "u'ar'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b = TextBlob(u\"بسيط هو أفضل من مجمع\")\n",
    "b.detect_language()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TextBlob(\"\")"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chinese_blob = TextBlob(u\"美丽优于丑陋\")\n",
    "chinese_blob.translate(from_lang=\"zh-CN\", to='en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TextBlob(\"\")"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "en_blob = TextBlob(u\"Simple is better than complex.\")\n",
    "en_blob.translate(to=\"es\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## spaCy\n",
    "\n",
    "Industrial strength NLP, in Python but with a strong Cython backend. Super fast. Licensing issue though. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2303, u'baseball', 4193, u'bat')\n"
     ]
    }
   ],
   "source": [
    "from __future__ import unicode_literals \n",
    "from spacy.en import English\n",
    "\n",
    "nlp = English()\n",
    "\n",
    "tokens = nlp(u'The man hit the building with the baseball bat.')\n",
    "\n",
    "baseball = tokens[7]\n",
    "print (baseball.orth, baseball.orth_, baseball.head.lemma, baseball.head.lemma_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-5.02773189545\n",
      "-8.16621112823\n",
      "-8.3605670929\n",
      "-3.07847452164\n",
      "-8.67186450958\n",
      "-5.23164892197\n",
      "-3.07847452164\n",
      "-9.61269950867\n",
      "-10.9683980942\n",
      "-3.17597317696\n"
     ]
    }
   ],
   "source": [
    "tokens = nlp(u'The man hit the building with the baseball bat.', parse=True)\n",
    "for token in tokens:\n",
    "    print token.prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## gensim\n",
    "\n",
    "Library for bag of words clustering - LSA and LDA. \n",
    "\n",
    "Also implements word2vec - Google's word vectorizer: something that was explored in a previous post."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}