Ar-Ar.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Intro. to Snorkel: Extracting Spouse Relations from the News\n",
    "\n",
    "In this tutorial, we will walk through the process of using `Snorkel` to identify mentions of spouses in a corpus of news articles. The tutorial is broken up into 3 notebooks, each covering a step in the pipeline:\n",
    "1. Preprocessing\n",
    "2. Training\n",
    "3. Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part I: Preprocessing\n",
    "\n",
    "In this notebook, we preprocess several documents using `Snorkel` utilities, parsing them into a simple hierarchy of component parts of our input data, which we refer to as _contexts_. We'll also create _candidates_ out of these contexts, which are the objects we want to classify, in this case, possible mentions of spouses. Finally, we'll load some gold labels for evaluation.\n",
    "\n",
    "All of this preprocessed input data is saved to a database.  (Connection strings can be specified by setting the `SNORKELDB` environment variable.  In Snorkel, if no database is specified, then a SQLite database at `./snorkel.db` is created by default--so no setup is needed here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initializing a `SnorkelSession`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we initialize a `SnorkelSession`, which manages a connection to a database automatically for us, and will enable us to save intermediate results.  If we don't specify any particular database (see commented-out code below), then it will automatically create a SQLite database in the background for us:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "os.environ['SNORKELDB'] = 'postgres://localhost/snorkel-intro'\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()\n",
    "\n",
    "# Here, we just set how many documents we'll process for automatic testing- you can safely ignore this!\n",
    "n_docs = 500 if 'CI' in os.environ else 2591"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the Corpus\n",
    "\n",
    "Next, we load and pre-process the corpus of documents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring a `DocPreprocessor`\n",
    "\n",
    "We'll start by defining a `TSVDocPreprocessor` class to read in the documents, which are stored in a tab-seperated value format as pairs of document names and text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.parser import GDDDocPreprocessor\n",
    "\n",
    "doc_preprocessor = GDDDocPreprocessor('data/sentences_nlp352', max_docs=n_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running a `CorpusParser`\n",
    "\n",
    "We'll use [Spacy](https://spacy.io/), an NLP preprocessing tool, to split our documents into sentences and tokens, and provide named entity annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n",
      "CPU times: user 6min 1s, sys: 1min 45s, total: 7min 47s\n",
      "Wall time: 17min 1s\n"
     ]
    }
   ],
   "source": [
    "from snorkel.parser.gdd_parser import Gdd\n",
    "from snorkel.parser import CorpusParser\n",
    "\n",
    "corpus_parser = CorpusParser(parser=Gdd())\n",
    "%time corpus_parser.apply(doc_preprocessor, count=n_docs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use simple database queries (written in the syntax of [SQLAlchemy](http://www.sqlalchemy.org/), which Snorkel uses) to check how many documents and sentences were parsed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Documents:', 475L)\n",
      "('Sentences:', 280651L)\n"
     ]
    }
   ],
   "source": [
    "from snorkel.models import Document, Sentence\n",
    "\n",
    "print(\"Documents:\", session.query(Document).count())\n",
    "print(\"Sentences:\", session.query(Sentence).count())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generating Candidates\n",
    "\n",
    "The next step is to extract _candidates_ from our corpus. A `Candidate` in Snorkel is an object for which we want to make a prediction. In this case, the candidates are pairs of people mentioned in sentences, and our task is to predict which pairs are described as married in the associated text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining a `Candidate` schema\n",
    "We now define the schema of the relation mention we want to extract (which is also the schema of the candidates).  This must be a subclass of `Candidate`, and we define it using a helper function. Here we'll define a binary _spouse relation mention_ which connects two `Span` objects of text.  Note that this function will create the table in the database backend if it does not exist:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "Measurement = candidate_subclass('Measurement', ['strat_name', 'value'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing a basic `CandidateExtractor`\n",
    "\n",
    "Next, we'll write a basic function to extract **candidate spouse relation mentions** from the corpus.  The [Spacy](https://spacy.io/) parser we used performs _named entity recognition_ for us.\n",
    "\n",
    "We will extract `Candidate` objects of the `Spouse` type by identifying, for each `Sentence`, all pairs of n-grams (up to 7-grams) that were tagged as people. (An n-gram is a span of text made up of n tokens.) We do this with three objects:\n",
    "\n",
    "* A `ContextSpace` defines the \"space\" of all candidates we even potentially consider; in this case we use the `Ngrams` subclass, and look for all n-grams up to 7 words long\n",
    "\n",
    "* A `Matcher` heuristically filters the candidates we use.  In this case, we just use a pre-defined matcher which looks for all n-grams tagged by Spacy as \"PERSON\". The keyword argument `longest_match_only` means that we'll skip n-grams contained in other n-grams.\n",
    "\n",
    "* A `CandidateExtractor` combines this all together!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Bajada Del Agrio Group', 'Bajo de La Carpa Formation', 'Candeleros Formation', 'Carrere Formation', 'Cerro Lisandro Formation']\n"
     ]
    }
   ],
   "source": [
    "from snorkel.candidates import Ngrams, CandidateExtractor\n",
    "from snorkel.matchers import DictionaryMatch, RegexMatchSpan\n",
    "\n",
    "ngrams         = Ngrams(n_max=7)\n",
    "#person_matcher = PersonMatcher(longest_match_only=True)\n",
    "strat_names = []\n",
    "with open(\"data/strat_names\") as fin:\n",
    "    for line in fin:\n",
    "        strat_names.append(line.strip().replace('\"', ''))\n",
    "print strat_names[:5]\n",
    "    \n",
    "stratname_matcher = DictionaryMatch(d=strat_names)\n",
    "measurement_matcher = RegexMatchSpan(rgx = '\\d+(?:\\.\\d*)?(?: ?[+ \\-±/Ϯ]+ *\\d*(?:\\.\\d*)* ?)*[MG]a')\n",
    "cand_extractor = CandidateExtractor(Measurement, [ngrams, ngrams], [stratname_matcher, measurement_matcher])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll split up the documents into train, development, and test splits; and collect the associated sentences.\n",
    "\n",
    "Note that we'll filter out a few sentences that mention more than five people. These lists are unlikely to contain spouses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/iross/miniconda2/envs/py2Env/lib/python2.7/site-packages/sqlalchemy/sql/compiler.py:624: SAWarning: Can't resolve label reference 'reverse(name)'; converting to text() (this warning may be suppressed after 10 occurrences)\n",
      "  util.ellipses_string(element.element))\n"
     ]
    }
   ],
   "source": [
    "from snorkel.models import Document\n",
    "from sqlalchemy.sql.expression import func\n",
    "docs = session.query(Document).order_by(\"reverse(name)\")\n",
    "# docs = session.query(Document).order_by(Document.name).all()\n",
    "\n",
    "train_sents = set()\n",
    "dev_sents   = set()\n",
    "test_sents  = set()\n",
    "\n",
    "for i, doc in enumerate(docs):\n",
    "    for s in doc.sentences:\n",
    "        if i % 10 == 9:\n",
    "            dev_sents.add(s)\n",
    "        elif i % 10 == 8:\n",
    "            test_sents.add(s)\n",
    "        else:\n",
    "            train_sents.add(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dev:  28233\n",
      "train:  220864\n",
      "test:  31554\n",
      "total:  280651\n"
     ]
    }
   ],
   "source": [
    "print \"dev: \", len(dev_sents)\n",
    "print \"train: \", len(train_sents)\n",
    "print \"test: \", len(test_sents)\n",
    "print \"total: \", len(dev_sents) + len(test_sents) + len(train_sents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we'll apply the candidate extractor to the three sets of sentences. The results will be persisted in the database backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n",
      "('Number of candidates:', 611L)\n",
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n",
      "('Number of candidates:', 206L)\n",
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n",
      "('Number of candidates:', 23L)\n",
      "CPU times: user 14min 2s, sys: 1min 54s, total: 15min 56s\n",
      "Wall time: 1h 24min 21s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "for i, sents in enumerate([train_sents, dev_sents, test_sents]):\n",
    "#for i, sents in enumerate([train_sents]):\n",
    "    cand_extractor.apply(sents, split=i)\n",
    "    print(\"Number of candidates:\", session.query(Measurement).filter(Measurement.split == i).count())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"jupyter-js-widgets\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Widget Javascript not detected.  It may not be installed or enabled properly.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "48562ea286fb45819153b4f370b950de"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "sv = SentenceNgramViewer(session.query(Measurement).filter(Measurement.split == 1), session, height=500)\n",
    "sv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from lib.init import *\n",
    "from lib.scoring import *\n",
    "from lib.lf_factories import *\n",
    "from snorkel.lf_helpers import *\n",
    "age_words = {'age', 'aged', 'dated', 'ages'}\n",
    "uncertainty_words = {'uncertainty', 'uncertainties'}\n",
    "\n",
    "\n",
    "LF_age_words = MatchTerms(name='age_words', terms=age_words, label=1, search='sentence').lf()\n",
    "\n",
    "LF_uncertainty_words = MatchTerms(name='uncertainty_words', terms=uncertainty_words, label=-1, search='sentence').lf()\n",
    "\n",
    "\n",
    "attributed_words = {'for', 'in'}\n",
    "LF_attributed_words = MatchTerms(name='attributed', terms=attributed_words, label=1, search='between').lf()\n",
    "LF_paren_between = MatchTerms(name='paren', terms={'-LRB-', '-RRB-'}, label=1, search='between').lf()\n",
    "\n",
    "def LF_Ar_Ar_mentioned(c):\n",
    "    text_splits = get_text_splits(c)\n",
    "    for text in text_splits:\n",
    "        if \"40Ar/39Ar\" in text: return 1\n",
    "    else:\n",
    "        return 0\n",
    "    \n",
    "def LF_Pb_Pb_mentioned(c):\n",
    "    text_splits = get_text_splits(c)\n",
    "    for text in text_splits:\n",
    "        if \"207Pb/206Pb\" in text: return 1\n",
    "    else:\n",
    "        return 0\n",
    "    \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_gold_labels\n",
    "L_gold_dev = load_gold_labels(session, annotator_name='iross', split=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "LF Score\n",
      "========================================\n",
      "Pos. class accuracy: 1.0\n",
      "Neg. class accuracy: 0.0\n",
      "Precision            0.474\n",
      "Recall               1.0\n",
      "F1                   0.643\n",
      "----------------------------------------\n",
      "TP: 18 | FP: 20 | TN: 0 | FN: 0\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#coverage(session, LF_Ar_Ar_mentioned, split=0)\n",
    "tp, fp, tn, fn = error_analysis(session, LF_age_words, split=1, gold=L_gold_dev)\n",
    "#SentenceNgramViewer(fp, session, n_per_page=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "resp = requests.get(\"https://macrostrat.org/api/v2/defs/strat_names?all\")\n",
    "stratz = resp.json()[\"success\"][\"data\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "macrostrat_ages = set()\n",
    "for strat in stratz:\n",
    "    macrostrat_ages.add((strat['strat_name'], str(strat['b_age']) + \" Ma\"))\n",
    "    macrostrat_ages.add((strat['strat_name'], str(strat['t_age']) + \" Ma\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{(u'Bilinguites Superbilinguis Marine Band', '323.2 Ma'),\n",
       " (u'Top Durham Low Main Coal', '323.2 Ma'),\n",
       " (u'Opeche Shale', '270.2333 Ma'),\n",
       " (u'Blackmount', '25.2 Ma'),\n",
       " (u'Sty Head Tarn', '458.4 Ma'),\n",
       " (u'Fuson Shale', '125 Ma'),\n",
       " (u'Wooly Dolomite', '1800 Ma'),\n",
       " (u'Tumbez Limestone', '458.4 Ma'),\n",
       " (u'Oregon Limestone', '470 Ma'),\n",
       " (u'Seafield Tower Limestone', '323.2 Ma'),\n",
       " (u'Yaouk Leucogranite', '427.4 Ma'),\n",
       " (u'Belmore Granite', '443.8 Ma'),\n",
       " (u'Marshall Green Coal', '323.2 Ma'),\n",
       " (u'Capita Shale', '33.9 Ma'),\n",
       " (u'Jordanhill New Blackband Ironstone', '323.2 Ma'),\n",
       " (u'Jacque Mountain Limestone', '312 Ma'),\n",
       " (u'Maidenhead', '0.781 Ma'),\n",
       " (u'Bow Breccia', '252.17 Ma'),\n",
       " (u'Moira', 'None Ma'),\n",
       " (u'Burnt Meadow Complex', '174.1 Ma'),\n",
       " (u'Orocopia Schist', '90.825 Ma'),\n",
       " (u'Gunder', '0.0117 Ma'),\n",
       " (u'Sandwick Fish Bed', '393.3 Ma'),\n",
       " (u'Nunniong Granodiorite', '427.4 Ma'),\n",
       " (u'Horton Creek', '346.7 Ma'),\n",
       " (u'Broubster Till', '0.0117 Ma'),\n",
       " (u'Roxana Silt', '0.15 Ma'),\n",
       " (u'Farmington Sandstone', '67.037 Ma'),\n",
       " (u'Vinemount', '423 Ma'),\n",
       " (u'Mount Buffalo Volcanics', '252.17 Ma'),\n",
       " (u'Clermont', '449.25 Ma'),\n",
       " (u'Ballyreagh Conglomerate', '323.2 Ma'),\n",
       " (u'Limekiln Coal', '323.2 Ma'),\n",
       " (u'Peppercorn', '423 Ma'),\n",
       " (u'Pitch Creek Volcanics', '393.3 Ma'),\n",
       " (u'Helmick', '33.9 Ma'),\n",
       " (u'Horse Cave Limestone', '330.9 Ma'),\n",
       " (u'Julian Schist', '201.3 Ma'),\n",
       " (u'Palisades Andesite', '2.588 Ma'),\n",
       " (u'Glen Falls', '548.725 Ma'),\n",
       " (u'Portage Canyon', '427.4 Ma'),\n",
       " (u'Lake Chamberlain Till', '0.0117 Ma'),\n",
       " (u'Little Mount Pleasant', '382.7 Ma'),\n",
       " (u'Prudhoe Bay', '201.3 Ma'),\n",
       " (u'Clayton Basalt', '5.333 Ma'),\n",
       " (u'Bimba Granite', '1600 Ma'),\n",
       " (u'Bannister Basalt', '28.1 Ma'),\n",
       " (u'Whey Whey', '1600 Ma'),\n",
       " (u'Sutter', '23.03 Ma'),\n",
       " (u'Brooks Mountain', '23.03 Ma'),\n",
       " (u'Turney Ranch', '102.625 Ma'),\n",
       " (u'Erath', '33.9 Ma'),\n",
       " (u'Wagait Granite', '1600 Ma'),\n",
       " (u'Canyon Range', '145 Ma'),\n",
       " (u'Colbeck Gneiss', '4000 Ma'),\n",
       " (u'James', '134.625 Ma'),\n",
       " (u'Markley Sandstone', '33.9 Ma'),\n",
       " (u'Gosforth Glacigenic', '0.126 Ma'),\n",
       " (u'Limestone Coal', '330.9 Ma'),\n",
       " (u'Knobs', '457.925 Ma'),\n",
       " (u'Kameruka Suite', '393.3 Ma'),\n",
       " (u'Durango Sand', '100.5 Ma'),\n",
       " (u'Crossgates Sandstone', '323.2 Ma'),\n",
       " (u'Neuqu\\xe9n', '100.5 Ma'),\n",
       " (u'Capitan', '272.3 Ma'),\n",
       " (u'Tim Bobbin Rock', '323.2 Ma'),\n",
       " (u'Hume Limestone', '427.4 Ma'),\n",
       " (u'Chainman Shale', '323.2 Ma'),\n",
       " (u'Waldoboro Granite', '358.9 Ma'),\n",
       " (u'Hawkhill', '56 Ma'),\n",
       " (u'Lone Mt Drift', '0.8705 Ma'),\n",
       " (u'Mynydd Myddfai Sandstone', '427.4 Ma'),\n",
       " (u'Kotcho', '372.2 Ma'),\n",
       " (u'Grainger', '353.9666 Ma'),\n",
       " (u'Shattered Range', '541 Ma'),\n",
       " (u'Dean Sandstone', '279.575 Ma'),\n",
       " (u'Crescent Hill Basalt', '38.675 Ma'),\n",
       " (u'Central Sandstone', 'None Ma'),\n",
       " (u'Comet Shale', '501 Ma'),\n",
       " (u'Wogatti Sandstone', '152.1 Ma'),\n",
       " (u'Ashfell Sandstone', '358.9 Ma'),\n",
       " (u'Moonbah Tonalite', '427.4 Ma'),\n",
       " (u'Mount Field Siltstone', '470 Ma'),\n",
       " (u'Ribble Sandstone', '145 Ma'),\n",
       " (u'Briscoe', 'None Ma'),\n",
       " (u'Leichhardt Supersuite', '272.3 Ma'),\n",
       " (u'Euroa Volcanic', '0 Ma'),\n",
       " (u'Curtis Diorite', '393.3 Ma'),\n",
       " (u'Sand Hill Gneiss', '541 Ma'),\n",
       " (u'Crab Lake', '0.15 Ma'),\n",
       " (u'Sangerville', '443.8 Ma'),\n",
       " (u'Camsell', '393.3 Ma'),\n",
       " (u'Mid Splint Coal', '323.2 Ma'),\n",
       " (u'Newby Sand', '33.9 Ma'),\n",
       " (u'Dounalt Limestone', '541 Ma'),\n",
       " (u'Gober Tongue', '86.3 Ma'),\n",
       " (u'Dongara Sandstone', '252.17 Ma'),\n",
       " (u'Maesmelyn Coal', '315.2 Ma'),\n",
       " (u'Ocotillo Conglomerate', '2.588 Ma'),\n",
       " (u'Southshore', '453 Ma'),\n",
       " (u'Neilead Semipelite', '1000 Ma'),\n",
       " (u'Dry Creek Shale', '490.725 Ma'),\n",
       " (u'McConnell Island', '170.3 Ma'),\n",
       " (u'St Ann and Vauclin', '15.97 Ma'),\n",
       " (u'Centennial School', '458.4 Ma'),\n",
       " (u'Warnes Sandstone', '1800 Ma'),\n",
       " (u'Sugarloaf Tuff', '72.1 Ma'),\n",
       " (u'Menteth', '382.7 Ma'),\n",
       " (u'Lesueur Sandstone', '199.3 Ma'),\n",
       " (u'San Juan', '2.588 Ma'),\n",
       " (u'Tolco Sand', '14.25 Ma'),\n",
       " (u'Segundo Lavas', '33.9 Ma'),\n",
       " (u'Tolga', '410.8 Ma'),\n",
       " (u'Specimen Hill Granite', '272.3 Ma'),\n",
       " (u'Cashaqua Shale', '379.925 Ma'),\n",
       " (u'Poughquag Quartzite', '513 Ma'),\n",
       " (u'Lebanon Granite', '382.7 Ma'),\n",
       " (u'Fifteen Mile Tank Conglomerate', '358.9 Ma'),\n",
       " (u'LaVale', '410.8 Ma'),\n",
       " (u'Folsomville', '312 Ma'),\n",
       " (u'Buckner', '163.5 Ma'),\n",
       " (u'Mandorah Shale', '541 Ma'),\n",
       " (u'Cabrach Sandstone', '419.2 Ma'),\n",
       " (u'Diamond City', '0.0117 Ma'),\n",
       " (u'Boswell', '310.5 Ma'),\n",
       " (u'Big Bore Granodiorite', '393.3 Ma'),\n",
       " (u'Owasco', '382.7 Ma'),\n",
       " (u'Nagog Pond Gneiss', '443.8 Ma'),\n",
       " (u'Teklanika', '49.85 Ma'),\n",
       " (u'Beauceville', '541 Ma'),\n",
       " (u'Yegua', '37.8 Ma'),\n",
       " (u'Madron', 'None Ma'),\n",
       " (u'Thirlwall Coal', '358.9 Ma'),\n",
       " (u'Middle House', '443.8 Ma'),\n",
       " (u'Gettel', '443.8 Ma'),\n",
       " (u'Henryhouse Limestone', '419.2 Ma'),\n",
       " (u'Clackamas Gravel', '2.588 Ma'),\n",
       " (u'Altanmoui Granite', '252.17 Ma'),\n",
       " (u'Kennebago', '485.4 Ma'),\n",
       " (u'Ladygrange Coal', '323.2 Ma'),\n",
       " (u'Sandend Harbour Limestone', '541 Ma'),\n",
       " (u'Mid Hosie Limestone', '323.2 Ma'),\n",
       " (u'Burragorang Claystone', '252.17 Ma'),\n",
       " (u'Longarm Quartzite', '578.6 Ma'),\n",
       " (u'Dalton Sandstone', '86.3 Ma'),\n",
       " (u'Utukok', '353.35 Ma'),\n",
       " (u'Koonenberry', '513 Ma'),\n",
       " (u'Taparoo Sandstone', '113 Ma'),\n",
       " (u'Funeral', '392.3666 Ma'),\n",
       " (u'Styx Coal Measures', '100.5 Ma'),\n",
       " (u'Montour Shale', '358.9 Ma'),\n",
       " (u'Under Dunnet Shale', '358.9 Ma'),\n",
       " (u'Rencher', '19.3225 Ma'),\n",
       " (u'Providence Limestone', '307.8 Ma'),\n",
       " (u'Roch Volcanic', '458.4 Ma'),\n",
       " (u'West Sole', '174.1 Ma'),\n",
       " (u'Gayndah', '237 Ma'),\n",
       " (u'Milford', '359.565 Ma'),\n",
       " (u'Alamogordo', '351.5 Ma'),\n",
       " (u'Labadie Limestone', '323.2 Ma'),\n",
       " (u'Marysville Sand', '0.0117 Ma'),\n",
       " (u'Hurlford Main Coal', '307 Ma'),\n",
       " (u'Dog Springs', '23.03 Ma'),\n",
       " (u'Shaftesbury Sandstone', '100.5 Ma'),\n",
       " (u'Owhena', '84.625 Ma'),\n",
       " (u'Winnipeg', '444.85 Ma'),\n",
       " (u'Bootu', '1800 Ma'),\n",
       " (u'Park Coal', '315.2 Ma'),\n",
       " (u'Mirrabooka', '93.9 Ma'),\n",
       " (u'Ernie Lagoon', '720 Ma'),\n",
       " (u'Fool Creek Conglomerate', '29.0667 Ma'),\n",
       " (u'Piles Creek Sandstone', 'None Ma'),\n",
       " (u'Elimeek Volcanics', '323.2 Ma'),\n",
       " (u'Knox Mountain Granite', '419.2 Ma'),\n",
       " (u'Dead River', '475.775 Ma'),\n",
       " (u'Barrow Range', '419.2 Ma'),\n",
       " (u'Cat Hill Gneiss', '1000 Ma'),\n",
       " (u'Chiwaukum Schist', '100.5 Ma'),\n",
       " (u'Ghangat Basalt', '393.3 Ma'),\n",
       " (u'Peterlee Sand And Gravel', '0.0117 Ma'),\n",
       " (u'Bethanga Gneissic Granite', '427.4 Ma'),\n",
       " (u'Ella Marble', '458.4 Ma'),\n",
       " (u'Irondale Coal', '252.17 Ma'),\n",
       " (u'Ettrain', '298.05 Ma'),\n",
       " (u'Gordonbrook Serpentinite', '323.2 Ma'),\n",
       " (u'Lower Stromness Flagstone', '393.3 Ma'),\n",
       " (u'Bonnor Suite', '323.2 Ma'),\n",
       " (u'Edgewood Limestone', '427.4 Ma'),\n",
       " (u'Adjie Granodiorite', '419.2 Ma'),\n",
       " (u'Crevison Sandstone', '100.5 Ma'),\n",
       " (u'Beech Granite', '1160 Ma'),\n",
       " (u'Keating Granodiorite', '252.17 Ma'),\n",
       " (u'Laframboise', '442.0666 Ma'),\n",
       " (u'Laggan Sandstone', '541 Ma'),\n",
       " (u'Carten Creek', '88.925 Ma'),\n",
       " (u'Tidbinbilla Quartzite', '427.4 Ma'),\n",
       " (u'Salsipuedes', '64.9 Ma'),\n",
       " (u'Mercy Sandstone Lentil', '100.5 Ma'),\n",
       " (u'Mareeba Suite', '272.3 Ma'),\n",
       " (u'Bandalup Ultramafics', '4000 Ma'),\n",
       " (u'Chipping Norton Limestone', '166.1 Ma'),\n",
       " (u'Brown Howe Tuff', '458.4 Ma'),\n",
       " (u'Ivy Point Shale', '393.3 Ma'),\n",
       " (u'Phoenix', '5.333 Ma'),\n",
       " (u'Cleia', '145 Ma'),\n",
       " (u'Santa Catalina', '4000 Ma'),\n",
       " (u'Bays of Maine Complex', '1600 Ma'),\n",
       " (u'Roseray', '166.1 Ma'),\n",
       " (u'Mixer Pond', '443.8 Ma'),\n",
       " (u'Hector', '14.3575 Ma'),\n",
       " (u'Minuma Range', '358.9 Ma'),\n",
       " (u'White Hill Road Limestone', '0.0117 Ma'),\n",
       " (u'Currawinya Conglomerate', '323.2 Ma'),\n",
       " (u'Caribe', '11.62 Ma'),\n",
       " (u'Leschenault', '0 Ma'),\n",
       " (u'Kushtaka', '42.925 Ma'),\n",
       " (u'Blackford Hill', '423 Ma'),\n",
       " (u'Fiddler Shell Bed', '323.2 Ma'),\n",
       " (u'Sustut', '33.9 Ma'),\n",
       " (u'Cape Grenville Volcanics', '252.17 Ma'),\n",
       " (u'Stanford', '163.5 Ma'),\n",
       " (u'Toongi', '427.4 Ma'),\n",
       " (u'Ruxton', '346.7 Ma'),\n",
       " (u'Burrego', '303.7 Ma'),\n",
       " (u'Dillard', '201.3 Ma'),\n",
       " (u'Boggy Hole Gneiss', '1600 Ma'),\n",
       " (u'Forbes Hill', '470 Ma'),\n",
       " (u'McAdams Lake', '382.7 Ma'),\n",
       " (u'Green Berry', '268.8 Ma'),\n",
       " (u'Council Trench', '252.17 Ma'),\n",
       " (u'New Accommodation Coal', '315.2 Ma'),\n",
       " (u'Heath', '323.2 Ma'),\n",
       " (u'Upper Stonesdale Limestone', '330.9 Ma'),\n",
       " (u'Carswell', '1500 Ma'),\n",
       " (u'Currabubula', '323.2 Ma'),\n",
       " (u'Cheitmore Limestone', '427.4 Ma'),\n",
       " (u\"O'Keefe\", '12.7 Ma'),\n",
       " (u'Millstone Point Sandstone', '358.9 Ma'),\n",
       " (u'Cornubian Catchments', '0 Ma'),\n",
       " (u'Campagrande', '125.7333 Ma'),\n",
       " (u'Stone Coal Bluff', '66 Ma'),\n",
       " (u'Angullong', '448.4 Ma'),\n",
       " (u'San Isabel Granite', '1000 Ma'),\n",
       " (u'Kaza', '547.267 Ma'),\n",
       " (u'Durst', '323.2 Ma'),\n",
       " (u'Pinnacle Hill Gneiss', '4000 Ma'),\n",
       " (u'Barton Dolerite', 'None Ma'),\n",
       " (u'Mount Laurel', '66 Ma'),\n",
       " (u'Solomonsville Beds', '2.588 Ma'),\n",
       " (u'Marra', '290.1 Ma'),\n",
       " (u'Beckworth Suite', '382.7 Ma'),\n",
       " (u'Noorinbee Granodiorite', '419.2 Ma'),\n",
       " (u'Faywood Rhyolite', '66 Ma'),\n",
       " (u'Kulele Limestone', '2500 Ma'),\n",
       " (u'Tinowon', '298.9 Ma'),\n",
       " (u'Britton Clay', '100.5 Ma'),\n",
       " (u'Auld Lange Syne', '174.1 Ma'),\n",
       " (u'Creggan Mooar', '458.4 Ma'),\n",
       " (u'Jemmys Point', '2.588 Ma'),\n",
       " (u'Gleann Charadail', '66 Ma'),\n",
       " (u'Ariki', '3.6 Ma'),\n",
       " (u'Rochdale', '479.625 Ma'),\n",
       " (u'Fort Worth Limestone', '100.5 Ma'),\n",
       " (u'Copper Harbor Conglomerate', '1063.75 Ma'),\n",
       " (u'Hamburg Dolomite', '497.85 Ma'),\n",
       " (u'Oxford Clay Top', '163.5 Ma'),\n",
       " (u'Pooranoo Metamorphics', '1800 Ma'),\n",
       " (u'Bettyhill', '541 Ma'),\n",
       " (u'Hazon Lee Coal', '323.2 Ma'),\n",
       " (u'Carse Creek Complex', '443.8 Ma'),\n",
       " (u'Dockum', '201.3 Ma'),\n",
       " (u'Duchesne River', '37.8 Ma'),\n",
       " (u'Yarrol', '298.9 Ma'),\n",
       " (u'Coedana Granite', '541 Ma'),\n",
       " (u'Muirs Creek Conglomerate', '298.9 Ma'),\n",
       " (u'Pendola Shale', '100.5 Ma'),\n",
       " (u'Dolyhir Limestone', '433.4 Ma'),\n",
       " (u'Jamtin Rhyolite', '298.9 Ma'),\n",
       " (u'Clutterbuck', '541 Ma'),\n",
       " (u'Mountain Lake', '458.4 Ma'),\n",
       " (u'Morley Lane Volcanic', '635 Ma'),\n",
       " (u'Cedar Fork Limestone', '346.7 Ma'),\n",
       " (u'Urannah Igneous Complex', '358.9 Ma'),\n",
       " (u'Huntersville', '389.1 Ma'),\n",
       " (u'Pine Hill Anhydrite', '163.5 Ma'),\n",
       " (u'Knife Lake', '2500 Ma'),\n",
       " (u'Reelsville Limestone', '328.5637 Ma'),\n",
       " (u'Crawford', '0.0117 Ma'),\n",
       " (u'Lueders Limestone', '278.3625 Ma'),\n",
       " (u'Upper Rowhurst Coal', '307 Ma'),\n",
       " (u'Fort Duncan', '513 Ma'),\n",
       " (u'Cookson', '501 Ma'),\n",
       " (u'Golden Ball Granite', '358.9 Ma'),\n",
       " (u'Tahkandit Limestone', '284.8167 Ma'),\n",
       " (u'Gardiner Sandstone', '1600 Ma'),\n",
       " (u'Eagle Peak Syenite', '2.588 Ma'),\n",
       " (u'Hoosac Schist', '513 Ma'),\n",
       " (u'Double Lakes', '1.7292 Ma'),\n",
       " (u'Ticonderoga Dolostone', '485.4 Ma'),\n",
       " (u'Bear Lake', '5.333 Ma'),\n",
       " (u'East Waterford', '443.8 Ma'),\n",
       " (u'Boroston', '358.9 Ma'),\n",
       " (u'Lenapah Limestone', '306.8572 Ma'),\n",
       " (u'Grapevine Gulch', '1600 Ma'),\n",
       " (u'Moonta Porphyry', '1600 Ma'),\n",
       " (u'Demopolis Chalk', 'None Ma'),\n",
       " (u'Strathaird Limestone', '93.9 Ma'),\n",
       " (u'Bakerstown Sandstone', '298.9 Ma'),\n",
       " (u'Morro', '362.225 Ma'),\n",
       " (u'Sonoita', '100.5 Ma'),\n",
       " (u'Esperanza', '1600 Ma'),\n",
       " (u'Milyakburra', '1800 Ma'),\n",
       " (u'Mason', '2.588 Ma'),\n",
       " (u'Bellingham Drift', '2.588 Ma'),\n",
       " (u'Clancy Granodiorite', '100.5 Ma'),\n",
       " (u'Sooley Volcanic', '443.8 Ma'),\n",
       " (u'Castle Reef Dolomite', '335 Ma'),\n",
       " (u'Stonesty Tuff', '443.8 Ma'),\n",
       " (u'Clarence River Supersuite', '298.9 Ma'),\n",
       " (u'Harrisons Opening', '541 Ma'),\n",
       " (u'Ucluth Volcanics', '229.5 Ma'),\n",
       " (u'Brownmore Sandstone', '346.7 Ma'),\n",
       " (u'Munmarlary Quartzite', '1600 Ma'),\n",
       " (u'Tar Springs', '323.2 Ma'),\n",
       " (u'Vick', '93.9 Ma'),\n",
       " (u'Rothlyn', '427.4 Ma'),\n",
       " (u'Stono Granite', '541 Ma'),\n",
       " (u'Dead Horse Metabasalt', '1800 Ma'),\n",
       " (u'Bowhan Sandstone', '298.9 Ma'),\n",
       " (u'Araby', '517.76 Ma'),\n",
       " (u'Yarra Delta', '2.588 Ma'),\n",
       " (u'Nehenta', '201.3 Ma'),\n",
       " (u'Menard Limestone', '324.38 Ma'),\n",
       " (u'Ojo Huelos', '237 Ma'),\n",
       " (u'Jueyes Lens', '66 Ma'),\n",
       " (u'Black Bay Gabbro', '1600 Ma'),\n",
       " (u'Whiskey Lake', '2500 Ma'),\n",
       " (u'Bergstrom', '83.6 Ma'),\n",
       " (u'Strodes Creek', '450 Ma'),\n",
       " (u'Mann Metamorphics', '1600 Ma'),\n",
       " (u'Brownstown Marl', '83.6 Ma'),\n",
       " (u'Cumnock Major Coal', '307 Ma'),\n",
       " (u'Mount Willoughby Limestone', '0.0117 Ma'),\n",
       " (u'Belle Rive Coal', '306 Ma'),\n",
       " (u'Ishpeming Basic Volcanics', '1600 Ma'),\n",
       " (u'Mount Whacogo Volcanics', '201.3 Ma'),\n",
       " (u'Hilltop Coal', '298.9 Ma'),\n",
       " (u'Asylum Granite', '419.2 Ma'),\n",
       " (u'Mooroongga', '541 Ma'),\n",
       " (u'Dugald Slates', 'None Ma'),\n",
       " (u'Echo Lake Granodiorite', '100.5 Ma'),\n",
       " (u'Cedar Breaks', '33.9 Ma'),\n",
       " (u'Amber', '443.8 Ma'),\n",
       " (u'Meadfoot', '410.8 Ma'),\n",
       " (u'Watab Amphibolite', '4000 Ma'),\n",
       " (u'Carland Sandstone', '358.9 Ma'),\n",
       " (u'Mount Stromlo Granite', '358.9 Ma'),\n",
       " (u'Edziza Spectrum', '8.2826 Ma'),\n",
       " (u'Yaringa', '419.2 Ma'),\n",
       " (u'Sinnipee', '450 Ma'),\n",
       " (u'Westford Schist', '4000 Ma'),\n",
       " (u'Breccia Creek Rhyolite', '323.2 Ma'),\n",
       " (u'Shelton', '456.7375 Ma'),\n",
       " (u'Harrell Shale', '372.2 Ma'),\n",
       " (u'Seco', '66 Ma'),\n",
       " (u'Gries Ranch', '23.03 Ma'),\n",
       " (u'Muirkirk Seven Fathom Coal', '315.2 Ma'),\n",
       " (u'Mount Augustus Sandstone', '1800 Ma'),\n",
       " (u'Mason Knob Monzogranite', '2800 Ma'),\n",
       " (u'Paynes Creek Basalt', '0 Ma'),\n",
       " (u'Lower Stanhope Coal', '315.2 Ma'),\n",
       " (u'Burdiehouse Limestone', '323.2 Ma'),\n",
       " (u'Thores', '477.7 Ma'),\n",
       " (u'Marigold Oolite', '323.2 Ma'),\n",
       " (u'Fortification Basalt', '2.588 Ma'),\n",
       " (u'Cooney Canyon', '33.9 Ma'),\n",
       " (u'Fleanor', '458.4 Ma'),\n",
       " (u'Pedmar', '100.5 Ma'),\n",
       " (u'Mirrabooka', '433.4 Ma'),\n",
       " (u'Bootheragandra', '358.9 Ma'),\n",
       " (u'Metasiete and Los Frailes', '64.9 Ma'),\n",
       " (u'Bee Hill Sandstone', '1600 Ma'),\n",
       " (u'Crousa Downs Gabbro', 'None Ma'),\n",
       " (u'Frezier Sandstone', '113 Ma'),\n",
       " (u'North Evans Limestone', '372.2 Ma'),\n",
       " (u'Moonmera Porphyritic Granodiorite', '298.9 Ma'),\n",
       " (u'Nuckulla Basalt', '1600 Ma'),\n",
       " (u'Wales', '635 Ma'),\n",
       " (u'Bath Range', '1800 Ma'),\n",
       " (u'Poplar Tank', '282 Ma'),\n",
       " (u'Adaminaby', '485.4 Ma'),\n",
       " (u'Eundynie', '47.8 Ma'),\n",
       " (u'Carefree', '2.588 Ma'),\n",
       " (u'Parker Coal', '306 Ma'),\n",
       " (u'Livengood Dome Chert', '439.825 Ma'),\n",
       " (u'Martindale Mudstone', '346.7 Ma'),\n",
       " (u'Forbes Hill Conglomerate', '470 Ma'),\n",
       " (u'Red River', '458.4 Ma'),\n",
       " (u'Coorong Creek Adamellite', '4000 Ma'),\n",
       " (u'Dennis Bridge Limestone', '307 Ma'),\n",
       " (u\"St Helen's Island\", '417.1 Ma'),\n",
       " (u'Moore', '315.2 Ma'),\n",
       " (u'Brand Hills', '485.4 Ma'),\n",
       " (u'Misenheimer Shale', '393.3 Ma'),\n",
       " (u'Falling Creek', '208.5 Ma'),\n",
       " (u'Wattaberri', '541 Ma'),\n",
       " (u'Greenwater Fan Alloformation', '5.333 Ma'),\n",
       " (u'Garden Creek Monzogranite', '3600 Ma'),\n",
       " (u'Shoo Fly Complex', '485.4 Ma'),\n",
       " (u'Agrio', '128.3 Ma'),\n",
       " (u'Tarring Neville Flint', '89.8 Ma'),\n",
       " (u'Canon de Caballeros', '419.2 Ma'),\n",
       " (u'Bellows Field', '0.0117 Ma'),\n",
       " (u'Little Charley Pond', '4000 Ma'),\n",
       " (u'Sunnyside', '100.5 Ma'),\n",
       " (u'Neds Beach Calcarenite', '0 Ma'),\n",
       " (u'Tinker Complex', '201.3 Ma'),\n",
       " (u'Strumble Head Volcanic', '467.3 Ma'),\n",
       " (u'Upper Lount Coal', '323.2 Ma'),\n",
       " (u'Coddington Till', '0.781 Ma'),\n",
       " (u'Dunham Salt', '163.5 Ma'),\n",
       " (u'Glady Fork Sandstone', '325.3071 Ma'),\n",
       " (u'Syracuse', '423 Ma'),\n",
       " (u'Berkshire Valley', '417.1 Ma'),\n",
       " (u'Woodnamoka Phyllite', '1000 Ma'),\n",
       " (u'Augusta', '456.5 Ma'),\n",
       " (u'Elphinstone Granite', '272.3 Ma'),\n",
       " (u'Pocket', '427.4 Ma'),\n",
       " (u'Mosher Dolomite', '3712 Ma'),\n",
       " (u'Turners Falls Sandstone', '182.7 Ma'),\n",
       " (u'Fachwen', '485.4 Ma'),\n",
       " (u'Wotjulum Porphyry', '2500 Ma'),\n",
       " (u'Ham-Scarfskerry Flagstone', '382.7 Ma'),\n",
       " (u'Talbingo Basalt', '393.3 Ma'),\n",
       " (u'Thorntonia Limestone', '501 Ma'),\n",
       " (u'Metchosin', '50.475 Ma'),\n",
       " (u'Boulton', '372.2 Ma'),\n",
       " (u'Eight Mile Dacite', '0.0117 Ma'),\n",
       " (u'Chapman Ridge Sandstone', '458.4 Ma'),\n",
       " (u'Mannie', '458.4 Ma'),\n",
       " (u'Jack Hills Limestone', 'None Ma'),\n",
       " (u'Smoky River', '66 Ma'),\n",
       " (u'Capron', '0.0117 Ma'),\n",
       " (u'Gettel', '427.4 Ma'),\n",
       " (u'Tower Wood Gravel', '33.9 Ma'),\n",
       " (u'Tamboon Suite', '419.2 Ma'),\n",
       " (u'Ditch Creek Siltstone', '86.3 Ma'),\n",
       " (u'Ballagan', '323.2 Ma'),\n",
       " (u'Scajaquada', '419.2 Ma'),\n",
       " (u'Organ Rock Shale', '272.3 Ma'),\n",
       " (u'Del Cuerto', '298.9 Ma'),\n",
       " (u'Paparoa', '70.5 Ma'),\n",
       " (u'Bow Island', '100.5 Ma'),\n",
       " (u'Relay Felsite', '541 Ma'),\n",
       " (u'Muredong Shale', 'None Ma'),\n",
       " (u'Maer Volcanics', '0 Ma'),\n",
       " (u'Fish Springs', '501 Ma'),\n",
       " (u'Pedlar River Charnockite Suite', '1000 Ma'),\n",
       " (u'Lower Garscadden Coal', '330.9 Ma'),\n",
       " (u'Featherstone Granodiorite', '272.3 Ma'),\n",
       " (u'Somerset Shale', '335 Ma'),\n",
       " (u'Munger Granite Porphyry', '4000 Ma'),\n",
       " (u'Tinkers Creek Marl', '0.0117 Ma'),\n",
       " (u'Shasta Bally Quartz Diorite', '145 Ma'),\n",
       " (u'Aramac Coal Measures', '298.9 Ma'),\n",
       " (u'Lawyer Creek', '5.333 Ma'),\n",
       " (u'Mooga Sandstone', '145 Ma'),\n",
       " (u'Cornell', '358.9 Ma'),\n",
       " (u'Eoline', '94.659 Ma'),\n",
       " (u'Siller Willie Coal', '330.9 Ma'),\n",
       " (u'Nameless Coal', '323.2 Ma'),\n",
       " (u'Loescher Sand', '49.85 Ma'),\n",
       " (u'Todilto Limestone', '163.5 Ma'),\n",
       " (u'Lyndhurst', '720 Ma'),\n",
       " (u'Tweed Valley', '0.0117 Ma'),\n",
       " (u'Uplands Province Volcanics', '2.588 Ma'),\n",
       " (u'Milltown Andesite', '17.6463 Ma'),\n",
       " (u'Mount Rankin', '358.9 Ma'),\n",
       " (u'Northwest Grampian Granitic Subsuite', '485.4 Ma'),\n",
       " (u'Panther Mountain', '386.8667 Ma'),\n",
       " (u'Woongarra Rhyolite', '2500 Ma'),\n",
       " (u'Browns Range Metamorphics', '2300 Ma'),\n",
       " (u'Memerambi Granite', '201.3 Ma'),\n",
       " (u'Mount Nicholas Coal Measures', '237 Ma'),\n",
       " (u'Salem Limestone', '323.2 Ma'),\n",
       " (u'Galesville Sandstone', '495.3667 Ma'),\n",
       " (u'Gibsons Gully Aplite', '298.9 Ma'),\n",
       " (u'Lansdowne Arkose', '2050 Ma'),\n",
       " (u'Greenvale Cove', '446.15 Ma'),\n",
       " (u'Salt Lake', '23.03 Ma'),\n",
       " (u'Hosston', '100.5 Ma'),\n",
       " (u'Nanaimo', '100.5 Ma'),\n",
       " (u'Barren Red', '307 Ma'),\n",
       " (u'Huckleberry Ridge ash', '5.333 Ma'),\n",
       " (u'Springtime Canyon Quartz Latite', '23.03 Ma'),\n",
       " (u'Terrigal', '252.17 Ma'),\n",
       " (u'Napeequa Schist', '201.3 Ma'),\n",
       " (u'Surprise Creek', '1600 Ma'),\n",
       " (u'Mucarabones Sand', '5.333 Ma'),\n",
       " (u'Cranstone', '443.8 Ma'),\n",
       " (u'Ree Burn', '438.5 Ma'),\n",
       " (u'Hardwood Mountain', '433.4 Ma'),\n",
       " (u'Little Fiery Coal', '307 Ma'),\n",
       " (u'Mungaroona Granodiorite', '4000 Ma'),\n",
       " (u'Rapide Granite', '1600 Ma'),\n",
       " (u'Cuttingsville', '126.1 Ma'),\n",
       " (u'Padthaway', '0 Ma'),\n",
       " (u'Wormwood', '443.8 Ma'),\n",
       " (u'Bimbimbie Granodiorite', '427.4 Ma'),\n",
       " (u'Juniper Ridge Sandstone', '100.5 Ma'),\n",
       " (u'Trenton', '445.2 Ma'),\n",
       " (u'Wangkonda', '541 Ma'),\n",
       " (u'Polelle', '2500 Ma'),\n",
       " (u'Imina', '433.4 Ma'),\n",
       " (u'Antelope Flats', '272.3 Ma'),\n",
       " (u'Wehutty', '586.9 Ma'),\n",
       " (u'Piper Gulch Monzonite', '252.17 Ma'),\n",
       " (u'Granger Mountain', '287.6333 Ma'),\n",
       " (u'Stanton Rock Suite', '427.4 Ma'),\n",
       " (u'Livingston Conglomerate', '322.5 Ma'),\n",
       " (u'Hillsoak Bore Metamorphics', '1000 Ma'),\n",
       " (u'Buzzard', '372.2 Ma'),\n",
       " (u'Lomita Marl', '2.588 Ma'),\n",
       " (u'Mazatzal Quartzite', '1000 Ma'),\n",
       " (u'Opeche Shale', '282 Ma'),\n",
       " (u'Foxs Camp', '419.2 Ma'),\n",
       " (u'Shelton', '485.4 Ma'),\n",
       " (u'Beech Creek Limestone', '323.2 Ma'),\n",
       " (u'Meall Dearg', '1200 Ma'),\n",
       " (u'Fencepost limestone', '100.5 Ma'),\n",
       " (u'Adel Mountain Volcanics', '62.7 Ma'),\n",
       " (u'Kershaw Coal', '323.2 Ma'),\n",
       " (u'Moya', '298.9 Ma'),\n",
       " (u'Warrandyte Conglomerate', '427.4 Ma'),\n",
       " (u'Crooked Finger', '5.333 Ma'),\n",
       " (u'Montezuma', '694.5 Ma'),\n",
       " (u'Reno Sandstone', '501 Ma'),\n",
       " (u'Blanca', '5.333 Ma'),\n",
       " (u'Bolton limestone', '298.9 Ma'),\n",
       " (u'McAlester', '312 Ma'),\n",
       " (u'Monastery Creek Phosphorite', '501 Ma'),\n",
       " (u'Ballark Conglomerate', '145 Ma'),\n",
       " (u'Lower Pigeon House Coal', '323.2 Ma'),\n",
       " (u'Campbell Knob Suite', '358.9 Ma'),\n",
       " (u'Pinjian Chert Breccia', '2500 Ma'),\n",
       " (u'Robinson Canyon', '17.0875 Ma'),\n",
       " (u'Skonun', '18.6058 Ma'),\n",
       " (u'Sandal', '419.2 Ma'),\n",
       " (u'Demon Ridge Andesite Porphyry', '358.9 Ma'),\n",
       " (u'Wooroolin Granite', '298.9 Ma'),\n",
       " (u'Emerald Creek Supersuite', '272.3 Ma'),\n",
       " (u'Greywacke Conglomerate', '393.3 Ma'),\n",
       " (u'Upper Four Feet Coal', '307 Ma'),\n",
       " (u'Moss Side', '252.17 Ma'),\n",
       " (u'Benton Spring', '33.9 Ma'),\n",
       " (u'Leny Limestone And Slate', '485.4 Ma'),\n",
       " (u'Catskill', '365.72 Ma'),\n",
       " (u'Evans Creek Drift', '0.0117 Ma'),\n",
       " (u'Codell Sandstone', '90.825 Ma'),\n",
       " (u'Kennicott', '103.625 Ma'),\n",
       " (u'Prydz Complex', 'None Ma'),\n",
       " (u'Rebecca Sandstone', '445.2 Ma'),\n",
       " (u'Dead Cattle Gully Ignimbrite', '427.4 Ma'),\n",
       " (u'Henley Sandstone', '125 Ma'),\n",
       " (u'Kent Till', '0.0117 Ma'),\n",
       " (u'Minnie Point', '279.3 Ma'),\n",
       " (u'Top Hutton Coal', '307 Ma'),\n",
       " (u'Oatka Creek Shale', '382.7 Ma'),\n",
       " (u'Washington Irving Sandstone', '298.9 Ma'),\n",
       " (u'Glen Parva', '0.0117 Ma'),\n",
       " (u'Hulcross', '108 Ma'),\n",
       " (u'Rowe', '83.6 Ma'),\n",
       " (u'Scranton Mountain', '479.625 Ma'),\n",
       " (u'Mount Norgate Granite', '393.3 Ma'),\n",
       " (u'Farragon Volcanic', '1000 Ma'),\n",
       " (u'Boothby Orthogneiss', '2050 Ma'),\n",
       " (u'Boxwell Suite', '298.9 Ma'),\n",
       " (u'Almaden Suite', '298.9 Ma'),\n",
       " (u'Brassfield Limestone', '443.8 Ma'),\n",
       " (u'Lowana', '358.9 Ma'),\n",
       " (u'Rillito', '23.875 Ma'),\n",
       " (u'Greensleeves', '2800 Ma'),\n",
       " (u'Archer Fiord', '524 Ma'),\n",
       " (u'Penamnen Tuff', '458.4 Ma'),\n",
       " (u'Narrowleaf Leucogranite', '419.2 Ma'),\n",
       " (u'80 Yard Band Coal', '323.2 Ma'),\n",
       " (u'Lauchie', '157.3 Ma'),\n",
       " (u'Yilgatherra', '1400 Ma'),\n",
       " (u'Palomas', '5.333 Ma'),\n",
       " (u'Coa Sandstone', '419.2 Ma'),\n",
       " (u'Tau Trachyte', '0.0117 Ma'),\n",
       " (u'Merriland Tonalite', '393.3 Ma'),\n",
       " (u'Ridge Basin', '23.03 Ma'),\n",
       " (u'Mandata', '412.2 Ma'),\n",
       " (u'Residual Deposits', '66 Ma'),\n",
       " (u'Perry', '382.7 Ma'),\n",
       " (u'Wilbur Creek', '23.03 Ma'),\n",
       " (u'Thornhill Sandstone', '272.3 Ma'),\n",
       " (u'Bonaparte', '346.7 Ma'),\n",
       " (u'Falls Brook', '2500 Ma'),\n",
       " (u'Andrecito', '358.9 Ma'),\n",
       " (u'La Pena', '116 Ma'),\n",
       " (u'Telegraph Creek', '80.725 Ma'),\n",
       " (u'Isom', '33.9 Ma'),\n",
       " (u'Haywood Under Coal', '330.9 Ma'),\n",
       " (u'Butte Valley Basalt', '2.588 Ma'),\n",
       " (u'Bordeaux Pit', '0 Ma'),\n",
       " (u'Maneater Suite', '323.2 Ma'),\n",
       " (u'Anchor Silt', '1.806 Ma'),\n",
       " (u'Lambeth', '47.8 Ma'),\n",
       " (u'Solway Marine Band', '307 Ma'),\n",
       " (u'Hull Cove', '513 Ma'),\n",
       " (u'Freeport', '303.7 Ma'),\n",
       " (u'Carters', '458.4 Ma'),\n",
       " (u'Bitter Beck', '485.4 Ma'),\n",
       " (u'Sedgefield', '0.9778 Ma'),\n",
       " (u'Hines Tongue', '541 Ma'),\n",
       " (u'Gunter Sandstone', '482.8334 Ma'),\n",
       " (u'Evanston', '56 Ma'),\n",
       " (u'Canoe', '38.675 Ma'),\n",
       " (u'Abbey Head Bed', '346.7 Ma'),\n",
       " (u'Granite Park', '1600 Ma'),\n",
       " (u'Arkona', '386.45 Ma'),\n",
       " (u'Sunbury Shale', '346.7 Ma'),\n",
       " (u'McDermott', '100.5 Ma'),\n",
       " (u'Exley Microgranite', '443.8 Ma'),\n",
       " (u'Bottom Tilley Coal', '323.2 Ma'),\n",
       " (u'Lamington Volcanics', '15.97 Ma'),\n",
       " (u'Ingalls', '33.9 Ma'),\n",
       " (u'Anderson Sand', '66 Ma'),\n",
       " (u'Creagan', '541 Ma'),\n",
       " (u'Vaughan Springs Quartzite', '541 Ma'),\n",
       " (u'Lazy Bar Andesite', '541 Ma'),\n",
       " (u'Bakersville Intrusive Suite', '541 Ma'),\n",
       " (u'Ingham Granite Complex', '272.3 Ma'),\n",
       " (u'Benbulben Shale', '323.2 Ma'),\n",
       " (u'Hannacroix', '393.3 Ma'),\n",
       " (u'Sheepcote Valley Flint', '83.6 Ma'),\n",
       " (u'Cullen Supersuite', '2050 Ma'),\n",
       " (u'Lone Pine Gravel', '66 Ma'),\n",
       " (u'Quinault', '1.9439 Ma'),\n",
       " (u'Bustard', '1800 Ma'),\n",
       " (u'Horsehair Creek Tonalite', '393.3 Ma'),\n",
       " (u'Little Union Quartz Latite', '66 Ma'),\n",
       " (u'Rockford Granite', '358.9 Ma'),\n",
       " (u'Mulky', '312 Ma'),\n",
       " (u'Cheshire Quartzite', '468.5 Ma'),\n",
       " (u'Astika', 'None Ma'),\n",
       " (u'Hong Kong Chert', '4000 Ma'),\n",
       " (u'Balook', '5.333 Ma'),\n",
       " (u'Wherretts Chert', '458.4 Ma'),\n",
       " (u'Creag Meagaidh Psammite', '1000 Ma'),\n",
       " (u'Clackmannan Blackband', '323.2 Ma'),\n",
       " (u'West Beacon Sandstone', 'None Ma'),\n",
       " (u'Farmville Metagranite', '358.9 Ma'),\n",
       " (u'Melozi', '145 Ma'),\n",
       " (u'Sodus Shale', '427.4 Ma'),\n",
       " (u'Duffy', '0.0117 Ma'),\n",
       " (u'Queen Charlotte', '66 Ma'),\n",
       " (u'Bonneterre', '470 Ma'),\n",
       " (u'McHale Granite', '1600 Ma'),\n",
       " (u'Reef Ridge Shale', '2.588 Ma'),\n",
       " (u'Mount Stromlo Granite', '298.9 Ma'),\n",
       " (u'Deceit', '419.2 Ma'),\n",
       " (u'Hinesville Limestone', '332.05 Ma'),\n",
       " (u'Conover Ranch', '315.2 Ma'),\n",
       " (u'Loomat Granite', '393.3 Ma'),\n",
       " (u'Fjerritslev', '190.8 Ma'),\n",
       " (u'Wheeler Shale', '502.38 Ma'),\n",
       " (u'Hollymount', '5.333 Ma'),\n",
       " (u'Intricate Basalt', '5.333 Ma'),\n",
       " (u'Bunyeroo', '541 Ma'),\n",
       " (u'Saddle Mountain Diorite', '201.3 Ma'),\n",
       " (u'Magoffin', '317.25 Ma'),\n",
       " (u'Nonesuch', '1023 Ma'),\n",
       " (u'Alecs Coal', '323.2 Ma'),\n",
       " (u'Hindmarsh Clay', '2.588 Ma'),\n",
       " (u'Helmshore Grit', '315.2 Ma'),\n",
       " (u'Canon Rhyolite', '12.17 Ma'),\n",
       " (u'Llanbedrog Volcanic', '443.8 Ma'),\n",
       " (u'Housum', '470 Ma'),\n",
       " (u'Lodge Farm Silt And Clay', '0.0117 Ma'),\n",
       " (u'Virtuewell Coal', '315.2 Ma'),\n",
       " (u'Morgan', '306 Ma'),\n",
       " (u'Yangalla', '1000 Ma'),\n",
       " (u'Webster Springs Sandstone', '323.2 Ma'),\n",
       " (u'Island Creek Shale', '306 Ma'),\n",
       " (u'Coatbridge Musselband Ironstone', '307 Ma'),\n",
       " (u'Bardahessiagh', '443.8 Ma'),\n",
       " (u'Crag', '5.333 Ma'),\n",
       " (u'Moulton', '120.5 Ma'),\n",
       " (u'Chilmark', '145 Ma'),\n",
       " (u'Salem Point Shale', '272.3 Ma'),\n",
       " (u'Citadel', '504 Ma'),\n",
       " (u'Glascarnoch Psammite', '2500 Ma'),\n",
       " (u'Pant Mawr Sandstone', '358.9 Ma'),\n",
       " (u'Sphinx Conglomerate', '62.7 Ma'),\n",
       " (u'Hunton', '443.8 Ma'),\n",
       " (u'King Granodiorite', '1600 Ma'),\n",
       " (u'Lower Longcraig Limestone', '323.2 Ma'),\n",
       " (u'Horton Creek', '358.9 Ma'),\n",
       " (u'Bell Top', '23.03 Ma'),\n",
       " (u'Town Beach Diorite', '201.3 Ma'),\n",
       " (u'Acme Dolomite', '282 Ma'),\n",
       " (u'Maddox', '1600 Ma'),\n",
       " (u'Bullock Hole Intrusion', '2500 Ma'),\n",
       " (u'Levencorroch Mudstone', '208.5 Ma'),\n",
       " (u'Mount Simon Sandstone', '485.4 Ma'),\n",
       " (u'Archer Mountain Suite', '1000 Ma'),\n",
       " (u'Expedition Creek Granodiorite', '323.2 Ma'),\n",
       " (u'Benevola', '513 Ma'),\n",
       " (u'Star Peak', '201.3 Ma'),\n",
       " (u'Saddle Mountain Diorite', '145 Ma'),\n",
       " (u'Lower West Mills Marine Band', '323.2 Ma'),\n",
       " (u'Mason Conglomerate', '720 Ma'),\n",
       " (u'Burned Mountain Metarhyolite', '2500 Ma'),\n",
       " (u'Betsie Shale', '323.2 Ma'),\n",
       " (u'Angel', '139.8 Ma'),\n",
       " (u'Swauger', '1000 Ma'),\n",
       " (u'Clay Spur Bentonite', '93.9 Ma'),\n",
       " (u'Buffalo Wallow', '327.684 Ma'),\n",
       " (u'Abuillot', '48.62 Ma'),\n",
       " (u'Sawpit Granodiorite', '1600 Ma'),\n",
       " (u'Wylie', '0.0117 Ma'),\n",
       " (u'Quarry Arenite', '433.4 Ma'),\n",
       " (u'Porcupine Hills', '56 Ma'),\n",
       " (u'Tumulites Pseudobilinguis Marine Band', '330.9 Ma'),\n",
       " (u'Perry Creek', '419.2 Ma'),\n",
       " (u'Stribling', '410.8 Ma'),\n",
       " (u'Ringslade Clay', '33.9 Ma'),\n",
       " (u'Mangalo Schist', '2050 Ma'),\n",
       " (u'A-Bed Evaporite', '252.17 Ma'),\n",
       " (u'Inishtrahull Gneiss', '1600 Ma'),\n",
       " (u'Hard Bed Coal', '315.2 Ma'),\n",
       " (u'Chester', '332.05 Ma'),\n",
       " (u'Wamsutta', '323.2 Ma'),\n",
       " (u'Smelser Pass', '247.2 Ma'),\n",
       " (u'Sylamore Sandstone', '379.925 Ma'),\n",
       " (u'Ennugan Mountains Granite', '1800 Ma'),\n",
       " (u'Chequamegon Sandstone', '916 Ma'),\n",
       " (u'Port Sorell', '720 Ma'),\n",
       " (u'Clam Bank', '427.4 Ma'),\n",
       " (u'North Sea Coast Glacigenic', '0.0117 Ma'),\n",
       " (u'Henshaw', '433.4 Ma'),\n",
       " (u'Loveland Loess', '1.5145 Ma'),\n",
       " (u'Hurupua Limestone', '5.585 Ma'),\n",
       " (u'Tallassee Metaquartzite', '1600 Ma'),\n",
       " (u'Dunnottar-Crawton', '427.4 Ma'),\n",
       " (u'Ginninderra Porphyry', '423 Ma'),\n",
       " (u'Mount Godkin Granite', '2500 Ma'),\n",
       " (u'Valentine Limestone', '382.7 Ma'),\n",
       " (u'Glendon Limestone', '32.74 Ma'),\n",
       " (u'Benning Gabbro', '1600 Ma'),\n",
       " (u'Dulas Coal', '307 Ma'),\n",
       " (u'Zoroaster Complex', '1000 Ma'),\n",
       " (u'Presidio', '125 Ma'),\n",
       " (u'Ravalli', '1600 Ma'),\n",
       " (u'Rockslide', '501 Ma'),\n",
       " (u'Chwefri', '458.4 Ma'),\n",
       " (u'Bridgeman Hill', '513 Ma'),\n",
       " (u'South Wales Upper Coal Measures', '307 Ma'),\n",
       " (u'Bishop Brook Limestone', '393.3 Ma'),\n",
       " (u'Nelungaloo Volcanics', '470 Ma'),\n",
       " (u'Upham', '457.925 Ma'),\n",
       " (u'Beacon Hill Flags', '315.2 Ma'),\n",
       " (u'Kobau', '342.75 Ma'),\n",
       " (u'Elbrook', '513 Ma'),\n",
       " (u'Dodman', '393.3 Ma'),\n",
       " (u'Niobrara Chalk', '72.1 Ma'),\n",
       " (u'Maules Creek', '290.1 Ma'),\n",
       " (u'Oolong Granite', '419.2 Ma'),\n",
       " (u'Currawalla Shale', '458.4 Ma'),\n",
       " (u'Sinnipee', '458.4 Ma'),\n",
       " (u'Hare Fiord', '290.1 Ma'),\n",
       " (u'Pepper Shale', '94.659 Ma'),\n",
       " (u'Caenlochan Schist', '541 Ma'),\n",
       " (u'General Petroleum', '109.0938 Ma'),\n",
       " (u'Stevens Ridge', '5.333 Ma'),\n",
       " (u'Bourimbla Limestone', '458.4 Ma'),\n",
       " (u'Lidget Coal', '307 Ma'),\n",
       " (u'Irindina Gneiss', '513 Ma'),\n",
       " (u'Rowe Chalk', '83.6 Ma'),\n",
       " (u'Arcoona Quartzite', '541 Ma'),\n",
       " (u'Supplejack Dolostone', '1000 Ma'),\n",
       " (u'Dolgeville Limestone', '456.5 Ma'),\n",
       " (u'Kalorama', '0.0117 Ma'),\n",
       " (u'Sinsinawa', '443.8 Ma'),\n",
       " (u'Hong Kong Chert', '2500 Ma'),\n",
       " (u'Pertaoorrta', '1000 Ma'),\n",
       " (u'Clinch Sandstone', '433.4 Ma'),\n",
       " (u'Wadleigh Limestone', '357.05 Ma'),\n",
       " (u'Wartook Sandstone', '485.4 Ma'),\n",
       " (u'Hartgrove Limestone', '272.3 Ma'),\n",
       " (u'Kangaloon Sandstone', '272.3 Ma'),\n",
       " (u'Leichhardt Pocket Granite', '272.3 Ma'),\n",
       " (u'Fillmore Glen', '393.3 Ma'),\n",
       " (u'Frankfort', '449.75 Ma'),\n",
       " (u'Safford Canyon', '33.9 Ma'),\n",
       " (u'Greens Canyon Tuff', '23.03 Ma'),\n",
       " (u'Abbeville', '20.44 Ma'),\n",
       " (u'Thirlstane Basalt', '33.9 Ma'),\n",
       " (u'Sugar Bag Granite', '298.9 Ma'),\n",
       " (u'Astell Sandstone', '1600 Ma'),\n",
       " (u'Pilbara', '3600 Ma'),\n",
       " (u'Pinguicula', '1292.5 Ma'),\n",
       " (u'Mullynagowan Sandstone', '358.9 Ma'),\n",
       " (u'Big House Chalk', '100.5 Ma'),\n",
       " (u'Stagg Creek', '238.25 Ma'),\n",
       " (u'Harmon', '110.9166 Ma'),\n",
       " (u'Broadview Schist', '1800 Ma'),\n",
       " (u'Berg', '541 Ma'),\n",
       " (u'Latrobe Valley', '56 Ma'),\n",
       " (u'Cedar Bluffs Till', '0.0117 Ma'),\n",
       " (u'Cravatt', '393.3 Ma'),\n",
       " (u'Geuda Springs Shale', '298.9 Ma'),\n",
       " (u'Marlomumbah Granite', '2500 Ma'),\n",
       " (u'Falconer', '0.0117 Ma'),\n",
       " (u'Chatham Hill', '458.0865 Ma'),\n",
       " (u'Mahlac', '23.03 Ma'),\n",
       " (u'Moccasin Limestone', '456.8167 Ma'),\n",
       " (u'Point Reyes', '66 Ma'),\n",
       " (u'Coffman Conglomerate', '307.5 Ma'),\n",
       " (u'South Tyler', '95.55 Ma'),\n",
       " (u'Sniabar Limestone', '306 Ma'),\n",
       " (u'Lower Stinking Marine Band', '307 Ma'),\n",
       " (u'Rocky King Granodiorite', '427.4 Ma'),\n",
       " (u'Caddo', '100.5 Ma'),\n",
       " (u'Rooks River Granite', '358.9 Ma'),\n",
       " (u'Black Rabbit', 'None Ma'),\n",
       " (u'Nehenta', '237 Ma'),\n",
       " (u'Luscar', '108.3125 Ma'),\n",
       " (u'Concretionary Limestone', '272.3 Ma'),\n",
       " (u'Hollanders', '427.4 Ma'),\n",
       " (u'Siesta', '10.162 Ma'),\n",
       " (u'Alyangula', '2050 Ma'),\n",
       " (u'Coffee Mill Hammock Marl', '2.588 Ma'),\n",
       " (u'Sleeping Lion', '33.9 Ma'),\n",
       " (u'Malbaie', '391.9 Ma'),\n",
       " (u'Goose Egg', '282 Ma'),\n",
       " (u'Delegate Supersuite', 'None Ma'),\n",
       " (u'Waterton', '1000 Ma'),\n",
       " (u'Manchos Shale', 'None Ma'),\n",
       " (u'Dellwood Ignimbrite', '252.17 Ma'),\n",
       " (u'Soldier Summit', '56 Ma'),\n",
       " (u'Dry Fork', '237 Ma'),\n",
       " (u'Dry Hollow', '89.8 Ma'),\n",
       " (u'Ebagoola Granite', '427.4 Ma'),\n",
       " (u'Flattop Schist', '541 Ma'),\n",
       " (u'Musselburgh Splint Coal', '323.2 Ma'),\n",
       " (u'Waupaca Granite', '1600 Ma'),\n",
       " (u'Tisbury', '152.1 Ma'),\n",
       " (u'Herrick', '2.588 Ma'),\n",
       " (u'Ivy May Andesite', '66 Ma'),\n",
       " (u'Callina Supersuite', '3600 Ma'),\n",
       " (u'George River', '850 Ma'),\n",
       " (u'Sadlerochit', '298.9 Ma'),\n",
       " (u'Poudre Pass Rhyolite', '23.03 Ma'),\n",
       " (u'Derndang Granite', '419.2 Ma'),\n",
       " (u'Armsheugh Sand and Gravel', '0.781 Ma'),\n",
       " (u'Jack Hills', 'None Ma'),\n",
       " (u'Sand Wells', '125 Ma'),\n",
       " (u'Coolibah', '470 Ma'),\n",
       " (u'Tuscaloosa', '87.175 Ma'),\n",
       " (u'Utley Limestone', '33.9 Ma'),\n",
       " (u'Flume', '434.25 Ma'),\n",
       " (u'Tinkers Creek Marl', '2.588 Ma'),\n",
       " (u'Degonia Sandstone', '323.7363 Ma'),\n",
       " (u'Ryan Basalt', '4.0333 Ma'),\n",
       " (u'Ina Volcanics', '443.8 Ma'),\n",
       " (u'White Rabbit Diorite', '419.2 Ma'),\n",
       " (u'El Dorado Ridge Quartz Diorite', '66 Ma'),\n",
       " (u'Shellmound', '449.25 Ma'),\n",
       " (u'Hard Bed Band Coal', '315.2 Ma'),\n",
       " (u'Armadale Ball Coal', '323.2 Ma'),\n",
       " (u'Carbonado', '33.9 Ma'),\n",
       " (u'Clipper', '98.95 Ma'),\n",
       " (u'Triple', '541 Ma'),\n",
       " (u'Kiowa Shale', '109.875 Ma'),\n",
       " (u'Bonnor Creek Granite', '298.9 Ma'),\n",
       " (u'Groat Sandstone', '83.6 Ma'),\n",
       " (u'Loch Na Dal', '1000 Ma'),\n",
       " (u'Bridgewater', '2.588 Ma'),\n",
       " (u'Thornsgill Till', '0.0117 Ma'),\n",
       " (u'Mudong Metamorphics', '541 Ma'),\n",
       " (u'Forest Lodge', '541 Ma'),\n",
       " (u'Stone Corral Dolomite', '274.725 Ma'),\n",
       " (u'West Spring', '5.333 Ma'),\n",
       " (u'Chagrin Shale', '369.2 Ma'),\n",
       " (u'Warialda Sandstone', '145 Ma'),\n",
       " (u'Mount Martha Granodiorite', '358.9 Ma'),\n",
       " (u'Loughridge Sandstone', '458.4 Ma'),\n",
       " (u'Atherton', '5.1164 Ma'),\n",
       " (u'Quarry Conglomerate', '2.588 Ma'),\n",
       " (u'Vacy Ignimbrite', '315 Ma'),\n",
       " (u'Iron Creek Greensand', '34.3 Ma'),\n",
       " (u'Cape Spear', 'None Ma'),\n",
       " (u'Chino Creek Granite', '1600 Ma'),\n",
       " (u'Chepstow', '64.9 Ma'),\n",
       " (u'Umpqua', '56 Ma'),\n",
       " (u'Goat Gills', '485.4 Ma'),\n",
       " (u'Saline', '56 Ma'),\n",
       " (u'Filer Sandstone Lentil', '393.3 Ma'),\n",
       " (u'Vinalhaven Rhyolite', '427.4 Ma'),\n",
       " (u'Bullo River Sandstone', '541 Ma'),\n",
       " (u'Mount Max Granite', '323.2 Ma'),\n",
       " (u'Mount Sugarloaf Igneous Complex', '443.8 Ma'),\n",
       " (u'Chavez Canyon', '33.9 Ma'),\n",
       " (u'Wall Sandstone', '298.9 Ma'),\n",
       " (u'Rocky Bridge Granodiorite', '419.2 Ma'),\n",
       " (u'Cowlitz', '34.875 Ma'),\n",
       " (u'Aquinnah Conglomerate', '0.0117 Ma'),\n",
       " (u'Muzzy Ridge', '393.3 Ma'),\n",
       " (u'Grain Valley coal', '312 Ma'),\n",
       " (u'Ice River', '386.45 Ma'),\n",
       " (u'Madeline', '290.1 Ma'),\n",
       " (u'Main Range Volcanics', '15.97 Ma'),\n",
       " (u'Matapo Sandstone', '34.3 Ma'),\n",
       " (u'Grove Church Shale', '330.9 Ma'),\n",
       " (u'Meshik Volcanics', '23.03 Ma'),\n",
       " (u'Wash Creek Slate', '560.1265 Ma'),\n",
       " (u'Dalles', '23.03 Ma'),\n",
       " (u\"Tam O'Shanter Supersuite\", '485.4 Ma'),\n",
       " (u'Quitman', '100.5 Ma'),\n",
       " (u'Cooney Tuff', '23.03 Ma'),\n",
       " (u'Flat Top', '272.3 Ma'),\n",
       " (u'Douglas Island Volcanics', '100.5 Ma'),\n",
       " (u'Bull Canyon', '252.17 Ma'),\n",
       " (u'Griffith Canyon', '2.588 Ma'),\n",
       " (u'Reeves', '298.9 Ma'),\n",
       " (u'Llangibby Beds', '423 Ma'),\n",
       " (u'Cumberland Head Shale', '458.4 Ma'),\n",
       " (u'Onteora', '382.7 Ma'),\n",
       " (u'Ramp Creek', '343.5 Ma'),\n",
       " (u'Britannia Sandstone', '113 Ma'),\n",
       " (u'Con Bore Granite', '1400 Ma'),\n",
       " (u'Rockledge', '485.4 Ma'),\n",
       " (u'Lynchburg', '690.25 Ma'),\n",
       " (u'Brora Roof Bed', '163.5 Ma'),\n",
       " (u'Robertson', '190.8 Ma'),\n",
       " (u'Cerros Del Rio Basalt', '0.0117 Ma'),\n",
       " (u'Haystack Creek Gneiss', '541 Ma'),\n",
       " (u'Chararoo Granite', '1600 Ma'),\n",
       " (u'Keel Quartzite', '443.8 Ma'),\n",
       " (u'New Haven Arkose', '200.9667 Ma'),\n",
       " (u'Ashgill Mountain Creek Conglomerate', 'None Ma'),\n",
       " (u'Solor Church', '1600 Ma'),\n",
       " (u'Gaviota', '33.9 Ma'),\n",
       " (u'Holloway Prairie', '0.0117 Ma'),\n",
       " (u'Copper Creek Granodiorite', '66 Ma'),\n",
       " (u'Cedar Barn', '435.1 Ma'),\n",
       " (u'Freeling Heights Quartzite', '1600 Ma'),\n",
       " (u'Bishops Lodge', '23.03 Ma'),\n",
       " (u'Pine Mountain', '485.4 Ma'),\n",
       " (u'Capelaw Volcanic', '410.8 Ma'),\n",
       " (u'Kootznahoo', '5.333 Ma'),\n",
       " (u'Queensberry', '443.8 Ma'),\n",
       " (u'Weiragoo Granite', '4000 Ma'),\n",
       " (u'Hurley', '382.7 Ma'),\n",
       " (u'Muskiki', '89.8 Ma'),\n",
       " (u'Knapp', '346.7 Ma'),\n",
       " (u'Marianna Limestone', '29.086 Ma'),\n",
       " (u'Bobs Ridge Sandstone', '419.2 Ma'),\n",
       " (u'Kandie Tank Limestone', '501 Ma'),\n",
       " (u'Piedra Parada', '5.333 Ma'),\n",
       " (u'Red Cliff Hole', '93.9 Ma'),\n",
       " (u'Curlew Limestone', '310.98 Ma'),\n",
       " (u'Croton', '346.7 Ma'),\n",
       " (u'Middle Calcareous Grit', '163.5 Ma'),\n",
       " (u'Bowring Limestone', '298.9 Ma'),\n",
       " (u'Alger', '433.4 Ma'),\n",
       " (u'Glen Doe Psammite', '1000 Ma'),\n",
       " (u'Plummer Hollow', '443.8 Ma'),\n",
       " (u'Kendall Mountain', '470 Ma'),\n",
       " (u'Mingus', '306 Ma'),\n",
       " (u'Cocks', 'None Ma'),\n",
       " (u'Becancour River', '445.8333 Ma'),\n",
       " (u'Warby Volcanic', '272.3 Ma'),\n",
       " (u'Warrego Granite', '1800 Ma'),\n",
       " (u'Glen Granite', '298.9 Ma'),\n",
       " (u'Third Limestone', '330.9 Ma'),\n",
       " (u'Rothea Porphyry', '358.9 Ma'),\n",
       " (u'Juniper Ridge Conglomerate', '100.5 Ma'),\n",
       " (u'Novato Conglomerate', '139.8 Ma'),\n",
       " (u'Nespelem Silt', '0.0117 Ma'),\n",
       " (u'Wratten Igneous Suite', '358.9 Ma'),\n",
       " (u'Elko', '38.3833 Ma'),\n",
       " (u\"Cadell's Parrot Coal\", '330.9 Ma'),\n",
       " (u'Stewarts Point', '84.05 Ma'),\n",
       " (u'Kingswood Great Coal', '323.2 Ma'),\n",
       " (u'The Valley Tank Arenite', '387.7 Ma'),\n",
       " (u'Fourmile Limestone', '298.9 Ma'),\n",
       " (u'Lower Gasswater Coal', '330.9 Ma'),\n",
       " (u'Selinsgrove Limestone', '390.388 Ma'),\n",
       " (u'Murray Bridge Granite', '443.8 Ma'),\n",
       " (u'City of Melbourne Volcanics', '541 Ma'),\n",
       " (u'Skudiburgh', '166.1 Ma'),\n",
       " (u'Hareshaw Conglomerate', '393.3 Ma'),\n",
       " (u'Luce Gravel', '2.841 Ma'),\n",
       " (u'Snowpatch', '171.25 Ma'),\n",
       " (u'Barnt Green Volcanic', '477.7 Ma'),\n",
       " ...}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "macrostrat_ages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Coverage: 1.46% (3/206)\n"
     ]
    }
   ],
   "source": [
    "LF_distant_supervision = DistantSupervision(\"macrostrat\", kb=macrostrat_ages).lf()\n",
    "labeled = coverage(session, LF_distant_supervision, split=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('Number labeled:', 0)\n"
     ]
    }
   ],
   "source": [
    "labeled = []\n",
    "for c in session.query(Measurement).filter(Measurement.split == 1).all():\n",
    "    if LF_Ar_Ar_mentioned(c) != 0:\n",
    "        labeled.append(c)\n",
    "print(\"Number labeled:\", len(labeled))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "LFs = [LF_age_words, LF_Ar_Ar_mentioned, LF_paren_between, LF_Pb_Pb_mentioned, LF_uncertainty_words, LF_distant_supervision]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Applying the Labeling Functions\n",
    "\n",
    "Next, we need to actually run the LFs over all of our training candidates, producing a set of `Labels` and `LabelKeys` (just the names of the LFs) in the database.  We'll do this using the `LabelAnnotator` class, a UDF which we will again run with `UDFRunner`.  **Note that this will delete any existing `Labels` and `LabelKeys` for this candidate set.**  We start by setting up the class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import LabelAnnotator\n",
    "labeler = LabelAnnotator(lfs=LFs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we run the `labeler`. Note that we set a random seed for reproducibility, since some of the LFs involve random number generators. Again, this can be run in parallel, given an appropriate database like Postgres is being used:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n",
      "CPU times: user 5.73 s, sys: 403 ms, total: 6.13 s\n",
      "Wall time: 8.49 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<611x6 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 643 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.random.seed(1701)\n",
    "%time L_train = labeler.apply(split=0)\n",
    "L_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we've already created the labels (saved in the database), we can load them in as a sparse matrix here too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<611x6 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 643 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train = labeler.load_matrix(session, split=0)\n",
    "L_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the returned matrix is a special subclass of the `scipy.sparse.csr_matrix` class, with some special features which we demonstrate below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Measurement(Span(\"Warramunga Formation\", sentence=575187, chars=[105,124], words=[18,19]), Span(\"1860 Ma\", sentence=575187, chars=[89,95], words=[14,15]))"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.get_candidate(session, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LabelKey (LF_TERMS_age_words_[sentence|words]_TRUE)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.get_key(session, 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also view statistics about the resulting label matrix.\n",
    "\n",
    "* **Coverage** is the fraction of candidates that the labeling function emits a non-zero label for.\n",
    "* **Overlap** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a non-zero label for.\n",
    "* **Conflict** is the fraction candidates that the labeling function emits a non-zero label for and that another labeling function emits a *conflicting* non-zero label for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_age_words_[sentence|words]_TRUE</th>\n",
       "      <td>0</td>\n",
       "      <td>0.338789</td>\n",
       "      <td>0.188216</td>\n",
       "      <td>0.003273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_Ar_Ar_mentioned</th>\n",
       "      <td>1</td>\n",
       "      <td>0.073650</td>\n",
       "      <td>0.073650</td>\n",
       "      <td>0.003273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_paren_[between|words]_TRUE</th>\n",
       "      <td>2</td>\n",
       "      <td>0.631751</td>\n",
       "      <td>0.142390</td>\n",
       "      <td>0.001637</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_Pb_Pb_mentioned</th>\n",
       "      <td>3</td>\n",
       "      <td>0.004910</td>\n",
       "      <td>0.004910</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_uncertainty_words_[sentence|words]_FALSE</th>\n",
       "      <td>4</td>\n",
       "      <td>0.003273</td>\n",
       "      <td>0.003273</td>\n",
       "      <td>0.003273</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_DIST_SUPERVISION_macrostrat_TRUE</th>\n",
       "      <td>5</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   j  Coverage  Overlaps  \\\n",
       "LF_TERMS_age_words_[sentence|words]_TRUE           0  0.338789  0.188216   \n",
       "LF_Ar_Ar_mentioned                                 1  0.073650  0.073650   \n",
       "LF_TERMS_paren_[between|words]_TRUE                2  0.631751  0.142390   \n",
       "LF_Pb_Pb_mentioned                                 3  0.004910  0.004910   \n",
       "LF_TERMS_uncertainty_words_[sentence|words]_FALSE  4  0.003273  0.003273   \n",
       "LF_DIST_SUPERVISION_macrostrat_TRUE                5  0.000000  0.000000   \n",
       "\n",
       "                                                   Conflicts  \n",
       "LF_TERMS_age_words_[sentence|words]_TRUE            0.003273  \n",
       "LF_Ar_Ar_mentioned                                  0.003273  \n",
       "LF_TERMS_paren_[between|words]_TRUE                 0.001637  \n",
       "LF_Pb_Pb_mentioned                                  0.000000  \n",
       "LF_TERMS_uncertainty_words_[sentence|words]_FALSE   0.003273  \n",
       "LF_DIST_SUPERVISION_macrostrat_TRUE                 0.000000  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_train.lf_stats(session)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Fitting the Generative Model\n",
    "Now, we'll train a model of the LFs to estimate their accuracies. Once the model is trained, we can combine the outputs of the LFs into a single, noise-aware training label set for our extractor. Intuitively, we'll model the LFs by observing how they overlap and conflict with each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inferred cardinality: 2\n"
     ]
    }
   ],
   "source": [
    "from snorkel.learning import GenerativeModel\n",
    "\n",
    "gen_model = GenerativeModel()\n",
    "gen_model.train(L_train, epochs=100, decay=0.95, step_size=0.1 / L_train.shape[0], reg_param=1e-6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.41471946,  0.14784156,  0.80976555,  0.0822947 ,  0.0712936 ,\n",
       "        0.06498801])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_model.weights.lf_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now apply the generative model to the training candidates to get the noise-aware training label set. We'll refer to these as the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_marginals = gen_model.marginals(L_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll look at the distribution of the training marginals:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAD4hJREFUeJzt3X+s3Xddx/Hni3VMFCJdeqmlrXSYgnQklHhTiJA4WXQV\nQjoMWToTaMhI0QwCCTF0/CEY06RGgZggmCILNUFmE8BVmGAp6MQg5W52P9pRqaxjrd16YeqAmJl2\nb/+438Khv+733nPPPe2H5yM5Od/v5/v5nPO+n5y8+r2f+z3fpqqQJLXrGeMuQJI0Wga9JDXOoJek\nxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXFLxl0AwLJly2rNmjXjLkOSLiv33HPPd6tqYrZ+\nl0TQr1mzhqmpqXGXIUmXlSSP9Onn0o0kNc6gl6TGGfSS1DiDXpIaN2vQJ/mZJPuT3JfkYJI/7Nqv\nTrI3ybe656UDY25LciTJ4SQ3jPIHkCRdXJ8z+qeA11TVy4D1wMYkrwS2Afuqai2wr9snyTpgM3At\nsBH4SJIrRlG8JGl2swZ9zfhBt3tl9yhgE7Cra98F3NhtbwLuqKqnquph4AiwYUGrliT11muNPskV\nSQ4AJ4G9VfV1YHlVnei6PAYs77ZXAo8ODD/WtUmSxqBX0FfV6apaD6wCNiR56VnHi5mz/N6SbE0y\nlWRqenp6LkMlSXMwp2/GVtV/J/kKM2vvjydZUVUnkqxg5mwf4DiwemDYqq7t7NfaCewEmJyc9H8o\nlxqxZtvn5z326I7XLWAlOqPPVTcTSZ7bbT8L+A3gm8AeYEvXbQtwZ7e9B9ic5Kok1wBrgf0LXbgk\nqZ8+Z/QrgF3dlTPPAHZX1eeSfA3YneQW4BHgJoCqOphkN3AIOAXcWlWnR1O+JGk2swZ9Vd0PvPw8\n7d8Drr/AmO3A9qGrkyQNzW/GSlLjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS\n4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXO\noJekxhn0ktQ4g16SGmfQS1LjDHpJatysQZ9kdZKvJDmU5GCSd3bt709yPMmB7vHagTG3JTmS5HCS\nG0b5A0iSLm5Jjz6ngHdX1b1JngPck2Rvd+xDVfWng52TrAM2A9cCzwe+lORFVXV6IQuXJPUz6xl9\nVZ2oqnu77e8DDwErLzJkE3BHVT1VVQ8DR4ANC1GsJGnu5rRGn2QN8HLg613TO5Lcn+T2JEu7tpXA\nowPDjnHxfxgkSSPUO+iTPBv4NPCuqnoS+CjwQmA9cAL4wFzeOMnWJFNJpqanp+cyVJI0B72CPsmV\nzIT8J6vqMwBV9XhVna6qp4GP8ePlmePA6oHhq7q2n1BVO6tqsqomJyYmhvkZJEkX0eeqmwAfBx6q\nqg8OtK8Y6PYG4MFuew+wOclVSa4B1gL7F65kSdJc9Lnq5lXAm4AHkhzo2t4L3JxkPVDAUeBtAFV1\nMMlu4BAzV+zc6hU3kjQ+swZ9VX0VyHkO3XWRMduB7UPUJUlaIH4zVpIaZ9BLUuMMeklqnEEvSY0z\n6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINe\nkhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuNmDfokq5N8JcmhJAeT\nvLNrvzrJ3iTf6p6XDoy5LcmRJIeT3DDKH0CSdHF9zuhPAe+uqnXAK4Fbk6wDtgH7qmotsK/bpzu2\nGbgW2Ah8JMkVoyhekjS7WYO+qk5U1b3d9veBh4CVwCZgV9dtF3Bjt70JuKOqnqqqh4EjwIaFLlyS\n1M+c1uiTrAFeDnwdWF5VJ7pDjwHLu+2VwKMDw451bWe/1tYkU0mmpqen51i2JKmv3kGf5NnAp4F3\nVdWTg8eqqoCayxtX1c6qmqyqyYmJibkMlSTNQa+gT3IlMyH/yar6TNf8eJIV3fEVwMmu/TiwemD4\nqq5NkjQGfa66CfBx4KGq+uDAoT3Alm57C3DnQPvmJFcluQZYC+xfuJIlSXOxpEefVwFvAh5IcqBr\ney+wA9id5BbgEeAmgKo6mGQ3cIiZK3ZurarTC165JKmXWYO+qr4K5AKHr7/AmO3A9iHqkiQtEL8Z\nK0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS\n1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mN\nmzXok9ye5GSSBwfa3p/keJID3eO1A8duS3IkyeEkN4yqcElSP33O6D8BbDxP+4eqan33uAsgyTpg\nM3BtN+YjSa5YqGIlSXM3a9BX1d3AEz1fbxNwR1U9VVUPA0eADUPUJ0ka0jBr9O9Icn+3tLO0a1sJ\nPDrQ51jXJkkak/kG/UeBFwLrgRPAB+b6Akm2JplKMjU9PT3PMiRJs5lX0FfV41V1uqqeBj7Gj5dn\njgOrB7qu6trO9xo7q2qyqiYnJibmU4YkqYd5BX2SFQO7bwDOXJGzB9ic5Kok1wBrgf3DlShJGsaS\n2Tok+RRwHbAsyTHgfcB1SdYDBRwF3gZQVQeT7AYOAaeAW6vq9GhKlyT1MWvQV9XN52n++EX6bwe2\nD1OUJGnh+M1YSWqcQS9JjTPoJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWp\ncQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn\n0EtS4wx6SWqcQS9JjZs16JPcnuRkkgcH2q5OsjfJt7rnpQPHbktyJMnhJDeMqnBJUj9LevT5BPBh\n4K8G2rYB+6pqR5Jt3f57kqwDNgPXAs8HvpTkRVV1emHL/klrtn1+3mOP7njdAlYiSZeeWc/oq+pu\n4ImzmjcBu7rtXcCNA+13VNVTVfUwcATYsEC1SpLmYb5r9Mur6kS3/RiwvNteCTw60O9Y1yZJGpOh\n/xhbVQXUXMcl2ZpkKsnU9PT0sGVIki5gvkH/eJIVAN3zya79OLB6oN+qru0cVbWzqiaranJiYmKe\nZUiSZjPfoN8DbOm2twB3DrRvTnJVkmuAtcD+4UqUJA1j1qtuknwKuA5YluQY8D5gB7A7yS3AI8BN\nAFV1MMlu4BBwCrh11FfcSJIubtagr6qbL3Do+gv03w5sH6YoSdLC6XMdvTR2w3xXAvy+hH66eQsE\nSWqcQS9JjTPoJalxrtFL0pAu9ftteUYvSY0z6CWpcQa9JDXOoJekxhn0ktQ4g16SGmfQS1LjDHpJ\napxBL0mNM+glqXEGvSQ1zqCXpMYZ9JLUOINekhpn0EtS4wx6SWqcQS9JjTPoJalxBr0kNc6gl6TG\nGfSS1LglwwxOchT4PnAaOFVVk0muBv4GWAMcBW6qqv8arkxJ0nwtxBn9r1fV+qqa7Pa3Afuqai2w\nr9uXJI3JKJZuNgG7uu1dwI0jeA9JUk/DBn0BX0pyT5KtXdvyqjrRbT8GLD/fwCRbk0wlmZqenh6y\nDEnShQy1Rg+8uqqOJ3kesDfJNwcPVlUlqfMNrKqdwE6AycnJ8/aRJA1vqDP6qjrePZ8EPgtsAB5P\nsgKgez45bJGSpPmbd9An+bkkzzmzDfwm8CCwB9jSddsC3DlskZKk+Rtm6WY58NkkZ17nr6vqC0m+\nAexOcgvwCHDT8GVKkuZr3kFfVd8GXnae9u8B1w9TlCRp4fjNWElqnEEvSY0z6CWpcQa9JDXOoJek\nxhn0ktQ4g16SGmfQS1LjDHpJapxBL0mNM+glqXHD3o9e0kWs2fb5eY89uuN1C1iJfpp5Ri9JjTPo\nJalxBr0kNc6gl6TGGfSS1DiDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJekxhn0ktS4kQV9\nko1JDic5kmTbqN5HknRxIwn6JFcAfw78FrAOuDnJulG8lyTp4kZ1P/oNwJGq+jZAkjuATcChEb2f\nNDLD3FNeuhSMKuhXAo8O7B8DXjGi95LUCP+jltFIVS38iyZvBDZW1Vu7/TcBr6iqtw/02Qps7XZf\nDBwe4i2XAd8dYnyLnJNzOSfnck7OdTnNyQuqamK2TqM6oz8OrB7YX9W1/UhV7QR2LsSbJZmqqsmF\neK1WOCfnck7O5Zycq8U5GdVVN98A1ia5Jskzgc3AnhG9lyTpIkZyRl9Vp5K8HfgicAVwe1UdHMV7\nSZIublRLN1TVXcBdo3r9syzIElBjnJNzOSfnck7O1dycjOSPsZKkS4e3QJCkxl02QT/bLRWSXJfk\nf5Ic6B5/MI46F1Of20x083IgycEk/7TYNY5Dj8/K7w98Th5McjrJ1eOodbH0mJOfT/J3Se7rPitv\nGUedi6nHnCxN8tkk9yfZn+Sl46hzQVTVJf9g5g+6/wG8EHgmcB+w7qw+1wGfG3etl9icPJeZbyP/\nYrf/vHHXfSnMy1n9Xw98edx1j3tOgPcCf9xtTwBPAM8cd+1jnpM/Ad7Xbf8ysG/cdc/3cbmc0f/o\nlgpV9X/AmVsq/DTrMye/A3ymqr4DUFUnF7nGcZjrZ+Vm4FOLUtn49JmTAp6TJMCzmQn6U4tb5qLq\nMyfrgC8DVNU3gTVJli9umQvjcgn6891SYeV5+v1q92vW3ye5dnFKG5s+c/IiYGmSf0xyT5I3L1p1\n49P3s0KSnwU2Ap9ehLrGqc+cfBh4CfCfwAPAO6vq6cUpbyz6zMl9wG8DJNkAvICZL39edkZ2eeUY\n3MvMEsUPkrwW+Ftg7ZhrGrclwK8A1wPPAr6W5F+r6t/HW9Yl4/XAv1TVE+Mu5BJwA3AAeA3wS8De\nJP9cVU+Ot6yx2gH8WZIDzPzj92/A6fGWND+Xyxl9n1sqPFlVP+i27wKuTLJs8UpcdLPOCTNnKV+s\nqh9W1XeBu4GXLVJ949JnXs7YTPvLNtBvTt7CzDJfVdUR4GFm1qVb1TdT3lJV64E3M/O3i28vXokL\n53IJ+llvqZDkF7r1xTO/Zj0D+N6iV7p4+txm4k7g1UmWdMsUrwAeWuQ6F1uv228k+Xng15iZo9b1\nmZPvMPObH9069Iu5TEOtpz6Z8tzuGMBbgbsv199wLoulm7rALRWS/G53/C+ANwK/l+QU8L/A5ur+\nXN6iPnNSVQ8l+QJwP/A08JdV9eD4qh69np8VgDcA/1BVPxxTqYum55z8EfCJJA8AAd7T/RbYpJ5z\n8hJgV5ICDgK3jK3gIfnNWElq3OWydCNJmieDXpIaZ9BLUuMMeklqnEEvSY0z6CWpcQa9JDXOoJek\nxv0/YQ0+646N9a0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x20abbea90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.hist(train_marginals, bins=20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view the learned accuracy parameters, and other statistics about the LFs learned by the generative model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Accuracy</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.692263</td>\n",
       "      <td>0.6811</td>\n",
       "      <td>0.700959</td>\n",
       "      <td>0.477908</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.571684</td>\n",
       "      <td>0.6724</td>\n",
       "      <td>0.574182</td>\n",
       "      <td>0.389538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.829503</td>\n",
       "      <td>0.7267</td>\n",
       "      <td>0.828057</td>\n",
       "      <td>0.601149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.538060</td>\n",
       "      <td>0.6700</td>\n",
       "      <td>0.541519</td>\n",
       "      <td>0.364375</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.539177</td>\n",
       "      <td>0.6662</td>\n",
       "      <td>0.538237</td>\n",
       "      <td>0.365366</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.531344</td>\n",
       "      <td>0.6668</td>\n",
       "      <td>0.527172</td>\n",
       "      <td>0.359421</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Accuracy  Coverage  Precision    Recall\n",
       "0  0.692263    0.6811   0.700959  0.477908\n",
       "1  0.571684    0.6724   0.574182  0.389538\n",
       "2  0.829503    0.7267   0.828057  0.601149\n",
       "3  0.538060    0.6700   0.541519  0.364375\n",
       "4  0.539177    0.6662   0.538237  0.365366\n",
       "5  0.531344    0.6668   0.527172  0.359421"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gen_model.learned_lf_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the Model to Iterate on Labeling Functions\n",
    "\n",
    "Now that we have learned the generative model, we can stop here and use this to potentially debug and/or improve our labeling function set.  First, we apply the LFs to our development set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%\n",
      "\n"
     ]
    }
   ],
   "source": [
    "L_dev = labeler.apply_existing(split=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And finally, we get the score of the generative model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 0.564\n",
      "Neg. class accuracy: 0.563\n",
      "Precision            0.232\n",
      "Recall               0.564\n",
      "F1                   0.328\n",
      "----------------------------------------\n",
      "TP: 22 | FP: 73 | TN: 94 | FN: 17\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tp, fp, tn, fn = gen_model.error_analysis(session, L_dev, L_gold_dev)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpreting Generative Model Performance\n",
    "\n",
    "At this point, we should be getting an F1 score of around 0.4 to 0.5 on the development set, which is pretty good!  However, we should be very careful in interpreting this. Since we developed our labeling functions using this development set as a guide, and our generative model is composed of these labeling functions, we expect it to score very well here!  \n",
    "\n",
    "In fact, it is probably somewhat _overfit_ to this set. However this is fine, since in the next tutorial, we'll train a more powerful end extraction model which will generalize beyond the development set, and which we will evaluate on a _blind_ test set (i.e. one we never looked at during development)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Doing Some Error Analysis\n",
    "\n",
    "At this point, we might want to look at some examples in one of the error buckets. For example, one of the false negatives that we did not correctly label as true mentions.  To do this, we can again just use the `Viewer`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"jupyter-js-widgets\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "\n",
    "# NOTE: This if-then statement is only to avoid opening the viewer during automated testing of this notebook\n",
    "# You should ignore this!\n",
    "import os\n",
    "if 'CI' not in os.environ:\n",
    "    sv = SentenceNgramViewer(fn, session)\n",
    "else:\n",
    "    sv = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Widget Javascript not detected.  It may not be installed or enabled properly.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d53ba43407674a64a957145220842952"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can save the best model from the hyperparameter search with a custom name so that we can reload it later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "ename": "IndexError",
     "evalue": "list index out of range",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIndexError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-35-e46c48090d6b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_selected\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0msv\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0munion\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfn\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/iross/gdd_snorkel/snorkel/viewer.pyc\u001b[0m in \u001b[0;36mget_selected\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    239\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    240\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mget_selected\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 241\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcandidates\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_selected_cid\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    242\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    243\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIndexError\u001b[0m: list index out of range"
     ]
    }
   ],
   "source": [
    "c = sv.get_selected() if sv else list(fp.union(fn))[0]\n",
    "c"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can easily see the labels that the LFs gave to this candidate using simple ORM-enabled syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "c.labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also now explore some of the additional functionalities of the `lf_stats` method for our dev set LF labels, `L_dev`: we can plug in the gold labels that we have, and the accuracies that our generative model has learned:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "snorkel/annotations.py:129: RuntimeWarning: invalid value encountered in divide\n",
      "  ac = (tp+tn).astype(float) / (tp+tn+fp+fn)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>j</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "      <th>TP</th>\n",
       "      <th>FP</th>\n",
       "      <th>FN</th>\n",
       "      <th>TN</th>\n",
       "      <th>Empirical Acc.</th>\n",
       "      <th>Learned Acc.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_age_words_[sentence|words]_TRUE</th>\n",
       "      <td>0</td>\n",
       "      <td>0.184466</td>\n",
       "      <td>0.067961</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.473684</td>\n",
       "      <td>0.690666</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_Ar_Ar_mentioned</th>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.571471</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_paren_[between|words]_TRUE</th>\n",
       "      <td>2</td>\n",
       "      <td>0.339806</td>\n",
       "      <td>0.077670</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13</td>\n",
       "      <td>56</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.188406</td>\n",
       "      <td>0.835792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_Pb_Pb_mentioned</th>\n",
       "      <td>3</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.535848</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_TERMS_uncertainty_words_[sentence|words]_FALSE</th>\n",
       "      <td>4</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.530226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LF_DIST_SUPERVISION_macrostrat_TRUE</th>\n",
       "      <td>5</td>\n",
       "      <td>0.014563</td>\n",
       "      <td>0.009709</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.540153</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   j  Coverage  Overlaps  \\\n",
       "LF_TERMS_age_words_[sentence|words]_TRUE           0  0.184466  0.067961   \n",
       "LF_Ar_Ar_mentioned                                 1  0.000000  0.000000   \n",
       "LF_TERMS_paren_[between|words]_TRUE                2  0.339806  0.077670   \n",
       "LF_Pb_Pb_mentioned                                 3  0.000000  0.000000   \n",
       "LF_TERMS_uncertainty_words_[sentence|words]_FALSE  4  0.000000  0.000000   \n",
       "LF_DIST_SUPERVISION_macrostrat_TRUE                5  0.014563  0.009709   \n",
       "\n",
       "                                                   Conflicts  TP  FP  FN  TN  \\\n",
       "LF_TERMS_age_words_[sentence|words]_TRUE                 0.0  18  20   0   0   \n",
       "LF_Ar_Ar_mentioned                                       0.0   0   0   0   0   \n",
       "LF_TERMS_paren_[between|words]_TRUE                      0.0  13  56   0   0   \n",
       "LF_Pb_Pb_mentioned                                       0.0   0   0   0   0   \n",
       "LF_TERMS_uncertainty_words_[sentence|words]_FALSE        0.0   0   0   0   0   \n",
       "LF_DIST_SUPERVISION_macrostrat_TRUE                      0.0   0   3   0   0   \n",
       "\n",
       "                                                   Empirical Acc.  \\\n",
       "LF_TERMS_age_words_[sentence|words]_TRUE                 0.473684   \n",
       "LF_Ar_Ar_mentioned                                            NaN   \n",
       "LF_TERMS_paren_[between|words]_TRUE                      0.188406   \n",
       "LF_Pb_Pb_mentioned                                            NaN   \n",
       "LF_TERMS_uncertainty_words_[sentence|words]_FALSE             NaN   \n",
       "LF_DIST_SUPERVISION_macrostrat_TRUE                      0.000000   \n",
       "\n",
       "                                                   Learned Acc.  \n",
       "LF_TERMS_age_words_[sentence|words]_TRUE               0.690666  \n",
       "LF_Ar_Ar_mentioned                                     0.571471  \n",
       "LF_TERMS_paren_[between|words]_TRUE                    0.835792  \n",
       "LF_Pb_Pb_mentioned                                     0.535848  \n",
       "LF_TERMS_uncertainty_words_[sentence|words]_FALSE      0.530226  \n",
       "LF_DIST_SUPERVISION_macrostrat_TRUE                    0.540153  "
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_dev.lf_stats(session, L_gold_dev, gen_model.learned_lf_stats()['Accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that for labeling functions with low coverage, our learned accuracies are closer to our prior of 70% accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving our training labels\n",
    "\n",
    "Finally, we'll save the `training_marginals`, which are our **probabilistic training labels**, so that we can use them in the next tutorial to train our end extraction model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 611 marginals\n",
      "CPU times: user 588 ms, sys: 56.3 ms, total: 644 ms\n",
      "Wall time: 1.09 s\n"
     ]
    }
   ],
   "source": [
    "from snorkel.annotations import save_marginals\n",
    "%time save_marginals(session, L_train, train_marginals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, in Part III, we'll use these probabilistic training labels to train a deep neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# ==================== END OF  PART 2 ======================"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part III: Training an End Extraction Model\n",
    "\n",
    "In this final section of the tutorial, we'll use the noisy training labels we generated in the last tutorial part to train our end extraction model.\n",
    "\n",
    "For this tutorial, we will be training a Bi-LSTM, a state-of-the-art deep neural network implemented in [TensorFlow](https://www.tensorflow.org/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "\n",
    "# TO USE A DATABASE OTHER THAN SQLITE, USE THIS LINE\n",
    "# Note that this is necessary for parallel execution amongst other things...\n",
    "# os.environ['SNORKELDB'] = 'postgres:///snorkel-intro'\n",
    "\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We repeat our definition of the `Spouse` `Candidate` subclass:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_marginals\n",
    "\n",
    "train_marginals = load_marginals(session, split=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also reload the candidates:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_cands = session.query(Measurement).filter(Measurement.split == 0).order_by(Measurement.id).all()\n",
    "dev_cands   = session.query(Measurement).filter(Measurement.split == 1).order_by(Measurement.id).all()\n",
    "test_cands  = session.query(Measurement).filter(Measurement.split == 2).order_by(Measurement.id).all()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we load gold labels for evaluation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"jupyter-js-widgets\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Widget Javascript not detected.  It may not be installed or enabled properly.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ad344d0322d041febfb0a86f7dc0855a"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "sv = SentenceNgramViewer(session.query(Measurement).filter(Measurement.split == 2), session, height=600)\n",
    "sv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from snorkel.annotations import load_gold_labels\n",
    "\n",
    "L_gold_dev  = load_gold_labels(session, annotator_name='iross', split=1)\n",
    "L_gold_test = load_gold_labels(session, annotator_name='iross', split=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can setup our discriminative model. Here we specify the model and learning hyperparameters.\n",
    "\n",
    "They can also be set automatically using a search based on the dev set with a [GridSearch](https://github.com/HazyResearch/snorkel/blob/master/snorkel/learning/utils.py) object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<206x1 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 201 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "L_gold_dev"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[reRNN] Training model\n",
      "[reRNN] n_train=507  #epochs=30  batch size=256\n",
      "[reRNN] Epoch 0 (1.67s)\tAverage loss=0.667006\tDev F1=31.84\n",
      "[reRNN] Epoch 1 (3.45s)\tAverage loss=0.509570\tDev F1=31.84\n",
      "[reRNN] Epoch 2 (5.26s)\tAverage loss=0.476701\tDev F1=31.84\n",
      "[reRNN] Epoch 3 (7.22s)\tAverage loss=0.474843\tDev F1=31.84\n",
      "[reRNN] Epoch 4 (8.70s)\tAverage loss=0.471913\tDev F1=31.84\n",
      "[reRNN] Epoch 5 (10.47s)\tAverage loss=0.472037\tDev F1=31.84\n",
      "[reRNN] Epoch 6 (12.37s)\tAverage loss=0.470101\tDev F1=31.84\n",
      "[reRNN] Epoch 7 (14.54s)\tAverage loss=0.470124\tDev F1=31.84\n",
      "[reRNN] Epoch 8 (17.59s)\tAverage loss=0.469771\tDev F1=31.84\n",
      "[reRNN] Epoch 9 (20.74s)\tAverage loss=0.469212\tDev F1=31.84\n",
      "[reRNN] Epoch 10 (23.98s)\tAverage loss=0.469150\tDev F1=31.84\n",
      "[reRNN] Epoch 11 (26.45s)\tAverage loss=0.468392\tDev F1=31.84\n",
      "[reRNN] Epoch 12 (28.40s)\tAverage loss=0.467908\tDev F1=31.84\n",
      "[reRNN] Epoch 13 (30.90s)\tAverage loss=0.467424\tDev F1=31.84\n",
      "[reRNN] Epoch 14 (32.56s)\tAverage loss=0.466366\tDev F1=31.84\n",
      "[reRNN] Epoch 15 (35.16s)\tAverage loss=0.465878\tDev F1=31.84\n",
      "[reRNN] Epoch 16 (37.16s)\tAverage loss=0.465006\tDev F1=31.84\n",
      "[reRNN] Epoch 17 (39.00s)\tAverage loss=0.464229\tDev F1=31.84\n",
      "[reRNN] Epoch 18 (40.50s)\tAverage loss=0.463613\tDev F1=31.84\n",
      "[reRNN] Epoch 19 (42.28s)\tAverage loss=0.462789\tDev F1=31.84\n",
      "[reRNN] Epoch 20 (43.71s)\tAverage loss=0.462140\tDev F1=31.84\n",
      "[reRNN] Epoch 21 (45.17s)\tAverage loss=0.461622\tDev F1=31.84\n",
      "[reRNN] Epoch 22 (46.74s)\tAverage loss=0.461154\tDev F1=31.84\n",
      "[reRNN] Epoch 23 (48.29s)\tAverage loss=0.460774\tDev F1=31.84\n",
      "[reRNN] Model saved as <reRNN>\n",
      "[reRNN] Epoch 24 (51.61s)\tAverage loss=0.460279\tDev F1=31.84\n",
      "[reRNN] Epoch 25 (53.34s)\tAverage loss=0.459991\tDev F1=31.84\n",
      "[reRNN] Epoch 26 (55.21s)\tAverage loss=0.459377\tDev F1=31.84\n",
      "[reRNN] Epoch 27 (57.21s)\tAverage loss=0.459609\tDev F1=31.84\n",
      "[reRNN] Epoch 28 (59.91s)\tAverage loss=0.459387\tDev F1=31.84\n",
      "[reRNN] Epoch 29 (62.32s)\tAverage loss=0.459671\tDev F1=31.84\n",
      "[reRNN] Training done (62.58s)\n",
      "INFO:tensorflow:Restoring parameters from checkpoints/reRNN/reRNN-23\n",
      "[reRNN] Loaded model <reRNN>\n"
     ]
    }
   ],
   "source": [
    "from snorkel.learning.disc_models.rnn import reRNN\n",
    "\n",
    "train_kwargs = {\n",
    "    'lr':         0.01,\n",
    "    'dim':        50,\n",
    "    'n_epochs':   30,\n",
    "    'dropout':    0.25,\n",
    "    'print_freq': 1,\n",
    "    'max_sentence_length': 100\n",
    "}\n",
    "\n",
    "lstm = reRNN(seed=1701, n_threads=None)\n",
    "lstm.train(train_cands, train_marginals, X_dev=dev_cands, Y_dev=L_gold_dev, **train_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also get the candidates returned in sets (true positives, false positives, true negatives, false negatives) as well as a more detailed score report:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 1.0\n",
      "Neg. class accuracy: 0.0\n",
      "Precision            0.522\n",
      "Recall               1.0\n",
      "F1                   0.686\n",
      "----------------------------------------\n",
      "TP: 12 | FP: 11 | TN: 0 | FN: 0\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tp, fp, tn, fn = lstm.error_analysis(session, test_cands, L_gold_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that if this is the final test set that you will be reporting final numbers on, to avoid biasing results you should not inspect results.  However you can run the model on your _development set_ and, as we did in the previous part with the generative labeling function model, inspect examples to do error analysis.\n",
    "\n",
    "You can also improve performance substantially by increasing the number of training epochs!\n",
    "\n",
    "Finally, we can save the predictions of the model on the test set back to the database. (This also works for other candidate sets, such as unlabeled candidates.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 23 marginals\n"
     ]
    }
   ],
   "source": [
    "lstm.save_marginals(session, test_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<snorkel.learning.disc_models.rnn.re_rnn.reRNN at 0x1e5a20bd0>"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lstm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[reRNN] Model saved as <measurements.lstm>\n"
     ]
    }
   ],
   "source": [
    "lstm.save(\"measurements.lstm\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 0 has argument past max length for model:\t[arg ends at index 154; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 25 has argument past max length for model:\t[arg ends at index 188; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 26 has argument past max length for model:\t[arg ends at index 188; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 27 has argument past max length for model:\t[arg ends at index 188; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 28 has argument past max length for model:\t[arg ends at index 188; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 46 has argument past max length for model:\t[arg ends at index 154; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 47 has argument past max length for model:\t[arg ends at index 147; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 48 has argument past max length for model:\t[arg ends at index 141; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 75 has argument past max length for model:\t[arg ends at index 108; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 77 has argument past max length for model:\t[arg ends at index 112; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 78 has argument past max length for model:\t[arg ends at index 120; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 79 has argument past max length for model:\t[arg ends at index 102; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 85 has argument past max length for model:\t[arg ends at index 195; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 86 has argument past max length for model:\t[arg ends at index 132; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 87 has argument past max length for model:\t[arg ends at index 232; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 88 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 89 has argument past max length for model:\t[arg ends at index 132; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 90 has argument past max length for model:\t[arg ends at index 132; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 91 has argument past max length for model:\t[arg ends at index 158; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 92 has argument past max length for model:\t[arg ends at index 209; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 93 has argument past max length for model:\t[arg ends at index 209; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 94 has argument past max length for model:\t[arg ends at index 232; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 95 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 96 has argument past max length for model:\t[arg ends at index 209; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 97 has argument past max length for model:\t[arg ends at index 209; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 98 has argument past max length for model:\t[arg ends at index 209; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 99 has argument past max length for model:\t[arg ends at index 260; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 100 has argument past max length for model:\t[arg ends at index 260; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 101 has argument past max length for model:\t[arg ends at index 260; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 102 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 103 has argument past max length for model:\t[arg ends at index 260; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 104 has argument past max length for model:\t[arg ends at index 260; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 105 has argument past max length for model:\t[arg ends at index 260; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 106 has argument past max length for model:\t[arg ends at index 195; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 108 has argument past max length for model:\t[arg ends at index 232; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 109 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 110 has argument past max length for model:\t[arg ends at index 119; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 112 has argument past max length for model:\t[arg ends at index 158; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 113 has argument past max length for model:\t[arg ends at index 195; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 114 has argument past max length for model:\t[arg ends at index 180; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 115 has argument past max length for model:\t[arg ends at index 232; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 116 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 117 has argument past max length for model:\t[arg ends at index 180; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 118 has argument past max length for model:\t[arg ends at index 180; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 119 has argument past max length for model:\t[arg ends at index 180; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 120 has argument past max length for model:\t[arg ends at index 195; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 121 has argument past max length for model:\t[arg ends at index 101; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 122 has argument past max length for model:\t[arg ends at index 232; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 123 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 124 has argument past max length for model:\t[arg ends at index 119; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 125 has argument past max length for model:\t[arg ends at index 101; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 126 has argument past max length for model:\t[arg ends at index 158; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 127 has argument past max length for model:\t[arg ends at index 238; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 128 has argument past max length for model:\t[arg ends at index 238; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 129 has argument past max length for model:\t[arg ends at index 238; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 130 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 131 has argument past max length for model:\t[arg ends at index 238; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 132 has argument past max length for model:\t[arg ends at index 238; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 133 has argument past max length for model:\t[arg ends at index 238; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 134 has argument past max length for model:\t[arg ends at index 195; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 135 has argument past max length for model:\t[arg ends at index 174; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 136 has argument past max length for model:\t[arg ends at index 232; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 137 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 138 has argument past max length for model:\t[arg ends at index 174; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 139 has argument past max length for model:\t[arg ends at index 174; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 140 has argument past max length for model:\t[arg ends at index 174; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 141 has argument past max length for model:\t[arg ends at index 226; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 142 has argument past max length for model:\t[arg ends at index 226; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 143 has argument past max length for model:\t[arg ends at index 232; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 144 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 145 has argument past max length for model:\t[arg ends at index 226; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 146 has argument past max length for model:\t[arg ends at index 226; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 147 has argument past max length for model:\t[arg ends at index 226; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 148 has argument past max length for model:\t[arg ends at index 195; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 149 has argument past max length for model:\t[arg ends at index 185; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 150 has argument past max length for model:\t[arg ends at index 232; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 151 has argument past max length for model:\t[arg ends at index 273; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 152 has argument past max length for model:\t[arg ends at index 185; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 153 has argument past max length for model:\t[arg ends at index 185; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 154 has argument past max length for model:\t[arg ends at index 185; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 195 has argument past max length for model:\t[arg ends at index 206; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n",
      "snorkel/learning/disc_models/rnn/rnn_base.py:34: UserWarning: Candidate 196 has argument past max length for model:\t[arg ends at index 206; max len 100]\n",
      "  warnings.warn('\\t'.join([w.format(i), info]))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 206 marginals\n"
     ]
    }
   ],
   "source": [
    "lstm.save_marginals(session, dev_cands)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "========================================\n",
      "Scores (Un-adjusted)\n",
      "========================================\n",
      "Pos. class accuracy: 1.0\n",
      "Neg. class accuracy: 0.0\n",
      "Precision            0.522\n",
      "Recall               1.0\n",
      "F1                   0.686\n",
      "----------------------------------------\n",
      "TP: 12 | FP: 11 | TN: 0 | FN: 0\n",
      "========================================\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tp, fp, tn, fn = lstm.error_analysis(session, test_cands, L_gold_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{Measurement(Span(\"Nason Ridge Migmatitic Gneiss\", sentence=340986, chars=[50,78], words=[8,11]), Span(\"91 Ma\", sentence=340986, chars=[86,90], words=[14,15])),\n",
       " Measurement(Span(\"Nason Ridge Migmatitic Gneiss\", sentence=341369, chars=[195,223], words=[27,30]), Span(\"90 Ma\", sentence=341369, chars=[471,475], words=[71,72])),\n",
       " Measurement(Span(\"Napier Complex\", sentence=437020, chars=[165,178], words=[32,33]), Span(\"2400 Ma\", sentence=437020, chars=[251,257], words=[44,45])),\n",
       " Measurement(Span(\"Fish Canyon Tuff\", sentence=359600, chars=[64,79], words=[11,13]), Span(\"28 Ma\", sentence=359600, chars=[58,62], words=[9,10])),\n",
       " Measurement(Span(\"Chiwaukum Schist\", sentence=341325, chars=[59,74], words=[12,13]), Span(\"96 Ma\", sentence=341325, chars=[116,120], words=[20,21])),\n",
       " Measurement(Span(\"Nason Ridge Migmatitic Gneiss\", sentence=340960, chars=[230,258], words=[39,42]), Span(\"91 Ma\", sentence=340960, chars=[89,93], words=[14,15])),\n",
       " Measurement(Span(\"Nason Ridge Migmatitic Gneiss\", sentence=341244, chars=[93,121], words=[19,22]), Span(\"90.1 ± 1.5 Ma\", sentence=341244, chars=[210,222], words=[42,45])),\n",
       " Measurement(Span(\"Chiwaukum Schist\", sentence=340986, chars=[29,44], words=[5,6]), Span(\"91 Ma\", sentence=340986, chars=[86,90], words=[14,15])),\n",
       " Measurement(Span(\"Wernecke Supergroup\", sentence=494532, chars=[100,118], words=[17,18]), Span(\"1610 -- 1642 Ma\", sentence=494532, chars=[162,176], words=[25,28])),\n",
       " Measurement(Span(\"Quartet Group\", sentence=494532, chars=[84,96], words=[14,15]), Span(\"1610 -- 1642 Ma\", sentence=494532, chars=[162,176], words=[25,28])),\n",
       " Measurement(Span(\"Nason Ridge Migmatitic Gneiss\", sentence=341390, chars=[117,145], words=[21,24]), Span(\"91 Ma\", sentence=341390, chars=[98,102], words=[16,17])),\n",
       " Measurement(Span(\"Chiwaukum Schist\", sentence=341390, chars=[151,166], words=[26,27]), Span(\"91 Ma\", sentence=341390, chars=[98,102], words=[16,17]))}"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "require.undef('viewer');\n",
       "\n",
       "// NOTE: all elements should be selected using this.$el.find to avoid collisions with other Viewers\n",
       "\n",
       "define('viewer', [\"jupyter-js-widgets\"], function(widgets) {\n",
       "    var ViewerView = widgets.DOMWidgetView.extend({\n",
       "        render: function() {\n",
       "            this.cids   = this.model.get('cids');\n",
       "            this.nPages = this.cids.length;\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Insert the html payload\n",
       "            this.$el.append(this.model.get('html'));\n",
       "\n",
       "            // Initialize all labels from previous sessions\n",
       "            this.labels = this.deserializeDict(this.model.get('_labels_serialized'));\n",
       "            for (var i=0; i < this.nPages; i++) {\n",
       "                this.pid = i;\n",
       "                for (var j=0; j < this.cids[i].length; j++) {\n",
       "                    this.cxid = j;\n",
       "                    for (var k=0; k < this.cids[i][j].length; k++) {\n",
       "                        this.cid = k;\n",
       "                        if (this.cids[i][j][k] in this.labels) {\n",
       "                            this.markCurrentCandidate(false);\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "            }\n",
       "            this.pid  = 0;\n",
       "            this.cxid = 0;\n",
       "            this.cid  = 0;\n",
       "\n",
       "            // Enable button functionality for navigation\n",
       "            var that = this;\n",
       "            this.$el.find(\"#next-cand\").click(function() {\n",
       "                that.switchCandidate(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-cand\").click(function() {\n",
       "                that.switchCandidate(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-context\").click(function() {\n",
       "                that.switchContext(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-context\").click(function() {\n",
       "                that.switchContext(-1);\n",
       "            });\n",
       "            this.$el.find(\"#next-page\").click(function() {\n",
       "                that.switchPage(1);\n",
       "            });\n",
       "            this.$el.find(\"#prev-page\").click(function() {\n",
       "                that.switchPage(-1);\n",
       "            });\n",
       "            this.$el.find(\"#label-true\").click(function() {\n",
       "                that.labelCandidate(true, true);\n",
       "            });\n",
       "            this.$el.find(\"#label-false\").click(function() {\n",
       "                that.labelCandidate(false, true);\n",
       "            });\n",
       "\n",
       "            // Arrow key functionality\n",
       "            this.$el.keydown(function(e) {\n",
       "                switch(e.which) {\n",
       "                    case 74: // j\n",
       "                    that.switchCandidate(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 73: // i\n",
       "                    that.switchPage(-1);\n",
       "                    break;\n",
       "\n",
       "                    case 76: // l\n",
       "                    that.switchCandidate(1);\n",
       "                    break;\n",
       "\n",
       "                    case 75: // k\n",
       "                    that.switchPage(1);\n",
       "                    break;\n",
       "\n",
       "                    case 84: // t\n",
       "                    that.labelCandidate(true, true);\n",
       "                    break;\n",
       "\n",
       "                    case 70: // f\n",
       "                    that.labelCandidate(false, true);\n",
       "                    break;\n",
       "                }\n",
       "            });\n",
       "\n",
       "            // Show the first page and highlight the first candidate\n",
       "            this.$el.find(\"#viewer-page-0\").show();\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Get candidate selector for currently selected candidate, escaping id properly\n",
       "        getCandidate: function() {\n",
       "            return this.$el.find(\".\"+this.cids[this.pid][this.cxid][this.cid]);\n",
       "        },  \n",
       "\n",
       "        // Color the candidate correctly according to registered label, as well as set highlighting\n",
       "        markCurrentCandidate: function(highlight) {\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var tags = this.$el.find(\".\"+cid);\n",
       "\n",
       "            // Clear color classes\n",
       "            tags.removeClass(\"candidate-h\");\n",
       "            tags.removeClass(\"true-candidate\");\n",
       "            tags.removeClass(\"true-candidate-h\");\n",
       "            tags.removeClass(\"false-candidate\");\n",
       "            tags.removeClass(\"false-candidate-h\");\n",
       "            tags.removeClass(\"highlighted\");\n",
       "\n",
       "            if (highlight) {\n",
       "                if (cid in this.labels) {\n",
       "                    tags.addClass(String(this.labels[cid]) + \"-candidate-h\");\n",
       "                } else {\n",
       "                    tags.addClass(\"candidate-h\");\n",
       "                }\n",
       "            \n",
       "            // If un-highlighting, leave with first non-null coloring\n",
       "            } else {\n",
       "                var that = this;\n",
       "                tags.each(function() {\n",
       "                    var cids = $(this).attr('class').split(/\\s+/).map(function(item) {\n",
       "                        return parseInt(item);\n",
       "                    });\n",
       "                    cids.sort();\n",
       "                    for (var i in cids) {\n",
       "                        if (cids[i] in that.labels) {\n",
       "                            var label = that.labels[cids[i]];\n",
       "                            $(this).addClass(String(label) + \"-candidate\");\n",
       "                            $(this).removeClass(String(!label) + \"-candidate\");\n",
       "                            break;\n",
       "                        }\n",
       "                    }\n",
       "                });\n",
       "            }\n",
       "\n",
       "            // Extra highlighting css\n",
       "            if (highlight) {\n",
       "                tags.addClass(\"highlighted\");\n",
       "            }\n",
       "\n",
       "            // Classes for showing direction of relation\n",
       "            if (highlight) {\n",
       "                this.$el.find(\".\"+cid+\"-0\").addClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").addClass(\"right-candidate\");\n",
       "            } else {\n",
       "                this.$el.find(\".\"+cid+\"-0\").removeClass(\"left-candidate\");\n",
       "                this.$el.find(\".\"+cid+\"-1\").removeClass(\"right-candidate\");\n",
       "            }\n",
       "        },\n",
       "\n",
       "        // Cycle through candidates and highlight, by increment inc\n",
       "        switchCandidate: function(inc) {\n",
       "            var N = this.cids[this.pid].length\n",
       "            var M = this.cids[this.pid][this.cxid].length;\n",
       "            if (N == 0 || M == 0) { return false; }\n",
       "\n",
       "            // Clear highlighting from previous candidate\n",
       "            if (inc != 0) {\n",
       "                this.markCurrentCandidate(false);\n",
       "\n",
       "                // Increment the cid counter\n",
       "\n",
       "                // Move to next context\n",
       "                if (this.cid + inc >= M) {\n",
       "                    while (this.cid + inc >= M) {\n",
       "                        \n",
       "                        // At last context on page, halt\n",
       "                        if (this.cxid == N - 1) {\n",
       "                            this.cid = M - 1;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to next context\n",
       "                        } else {\n",
       "                            inc -= M - this.cid;\n",
       "                            this.cxid += 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = 0;\n",
       "                        }\n",
       "                    }\n",
       "\n",
       "                // Move to previous context\n",
       "                } else if (this.cid + inc < 0) {\n",
       "                    while (this.cid + inc < 0) {\n",
       "                        \n",
       "                        // At first context on page, halt\n",
       "                        if (this.cxid == 0) {\n",
       "                            this.cid = 0;\n",
       "                            inc = 0;\n",
       "                            break;\n",
       "                        \n",
       "                        // Increment to previous context\n",
       "                        } else {\n",
       "                            inc += this.cid + 1;\n",
       "                            this.cxid -= 1;\n",
       "                            M = this.cids[this.pid][this.cxid].length;\n",
       "                            this.cid = M - 1;\n",
       "                        }\n",
       "                    }\n",
       "                }\n",
       "\n",
       "                // Move within current context\n",
       "                this.cid += inc;\n",
       "            }\n",
       "            this.markCurrentCandidate(true);\n",
       "\n",
       "            // Push this new cid to the model\n",
       "            this.model.set('_selected_cid', this.cids[this.pid][this.cxid][this.cid]);\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Switch through contexts\n",
       "        switchContext: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "\n",
       "            // Iterate context on this page\n",
       "            var M = this.cids[this.pid].length;\n",
       "            if (this.cxid + inc < 0) {\n",
       "                this.cxid = 0;\n",
       "            } else if (this.cxid + inc >= M) {\n",
       "                this.cxid = M - 1;\n",
       "            } else {\n",
       "                this.cxid += inc;\n",
       "            }\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Switch through pages\n",
       "        switchPage: function(inc) {\n",
       "            this.markCurrentCandidate(false);\n",
       "            this.$el.find(\".viewer-page\").hide();\n",
       "            if (this.pid + inc < 0) {\n",
       "                this.pid = 0;\n",
       "            } else if (this.pid + inc > this.nPages - 1) {\n",
       "                this.pid = this.nPages - 1;\n",
       "            } else {\n",
       "                this.pid += inc;\n",
       "            }\n",
       "            this.$el.find(\"#viewer-page-\"+this.pid).show();\n",
       "\n",
       "            // Show pagination\n",
       "            this.$el.find(\"#page\").html(this.pid);\n",
       "\n",
       "            // Reset cid and set to first candidate\n",
       "            this.cid = 0;\n",
       "            this.cxid = 0;\n",
       "            this.switchCandidate(0);\n",
       "        },\n",
       "\n",
       "        // Label currently-selected candidate\n",
       "        labelCandidate: function(label, highlighted) {\n",
       "            var c    = this.getCandidate();\n",
       "            var cid  = this.cids[this.pid][this.cxid][this.cid];\n",
       "            var cl   = String(label) + \"-candidate\";\n",
       "            var clh  = String(label) + \"-candidate-h\";\n",
       "            var cln  = String(!label) + \"-candidate\";\n",
       "            var clnh = String(!label) + \"-candidate-h\";\n",
       "\n",
       "            // Toggle label highlighting\n",
       "            if (c.hasClass(cl) || c.hasClass(clh)) {\n",
       "                c.removeClass(cl);\n",
       "                c.removeClass(clh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(\"candidate-h\");\n",
       "                }\n",
       "                this.labels[cid] = null;\n",
       "                this.send({event: 'delete_label', cid: cid});\n",
       "            } else {\n",
       "                c.removeClass(cln);\n",
       "                c.removeClass(clnh);\n",
       "                if (highlighted) {\n",
       "                    c.addClass(clh);\n",
       "                } else {\n",
       "                    c.addClass(cl);\n",
       "                }\n",
       "                this.labels[cid] = label;\n",
       "                this.send({event: 'set_label', cid: cid, value: label});\n",
       "            }\n",
       "\n",
       "            // Set the label and pass back to the model\n",
       "            this.model.set('_labels_serialized', this.serializeDict(this.labels));\n",
       "            this.touch();\n",
       "        },\n",
       "\n",
       "        // Serialization of hash maps, because traitlets Dict doesn't seem to work...\n",
       "        serializeDict: function(d) {\n",
       "            var s = [];\n",
       "            for (var key in d) {\n",
       "                s.push(key+\"~~\"+d[key]);\n",
       "            }\n",
       "            return s.join();\n",
       "        },\n",
       "\n",
       "        // Deserialization of hash maps\n",
       "        deserializeDict: function(s) {\n",
       "            var d = {};\n",
       "            var entries = s.split(/,/);\n",
       "            var kv;\n",
       "            for (var i in entries) {\n",
       "                kv = entries[i].split(/~~/);\n",
       "                if (kv[1] == \"true\") {\n",
       "                    d[kv[0]] = true;\n",
       "                } else if (kv[1] == \"false\") {\n",
       "                    d[kv[0]] = false;\n",
       "                }\n",
       "            }\n",
       "            return d;\n",
       "        },\n",
       "    });\n",
       "\n",
       "    return {\n",
       "        ViewerView: ViewerView\n",
       "    };\n",
       "});\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Widget Javascript not detected.  It may not be installed or enabled properly.\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ac0231ad42b242b4ba2c54f68721b38b"
      }
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from snorkel.viewer import SentenceNgramViewer\n",
    "sv = SentenceNgramViewer(tp, session, height=600)\n",
    "sv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "snorkel.viewer.SentenceNgramViewer"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(sv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}