WIP: Adding SRL

README.md

@@ -32,7 +32,8 @@ An application that identifies the part of speech (e.g. verb + tense, noun + num
 in plain text.
   - [illinois-ner](ner/README.md)
 An application that identifies named entities in plain text according to two different sets of categories. 
-
+  - [illinois-srl](srl/README.md)
+An application to annotate natural language sentences with semantic roles. 
 
 ## Using each library programmatically 
 

pom.xml

@@ -18,6 +18,7 @@
         <module>pos</module>
         <module>chunker</module>
         <module>ner</module>
+        <module>srl</module>
         <module>corpusreaders</module>
         <module>lbjava-nlp-tools</module>
     </modules>

srl/CHANGELOG

@@ -0,0 +1,55 @@
+Version 3.0.73
+Moved to the super-project and changed the versioning to the super-project versioning
+
+Version 5.1.12
+Added Windows support (including access to non-Gurobi solver)
+
+Version 5.1.4
+Switched entirely to illinois-sl for structured prediction (removed JLIS traces)
+Using the latest AnnotatorService from illinois-core-utilities for both Curator & pipeline annotation
+Major cleaning up
+
+Version 5.1
+Added JUnit tests
+Removed unnecessary dependencies
+Switched to illinois-nlp-pipeline-0.1.2
+Minor fixes
+
+Version 5.0
+Standalone SRL using illinois-nlp-pipeline
+
+Version 4.1.1
+Switched to edison-0.7.1 and LBJava-1.0
+Added dependency to illinois-common-resources
+
+Version 4.1
+Various bugfixes
+
+Version 4.0.2
+Updated inference dependency to latest version and modified inference
+code accordingly.
+
+Version 4.0.1
+Removed duplicate code from JLIS-core and moved to IllinoisSL. Minor edits.
+
+Version 4.0
+A complete rewrite of the SRL. Includes predicate and sense detectors,
+new constraints and a memory footprint of only 3GB.
+
+Version 3.0.3
+Minor bugfixes. Uses edison v0.2.9
+
+Version 3.0.2
+Added an option to trim leading prepositions from arguments.
+
+Revamped the training mechanism to train using LBJ's BatchTrainer in
+the code. This allows manual lexicon handling, which reduces the
+memory requirements by nearly 40 percent.
+
+Version 3.0.1
+Minor bugfix
+
+Version 3.0
+A complete Java based re-implementation of the Illinois SRL from
+Punyakanok 2008. This version uses LBJ to train classifiers and
+for performing inference with a home-brewed beam search.

srl/README.md

@@ -0,0 +1,49 @@
+# illinois-srl: Semantic Role Labeler
+
+### Running
+You can use the **illinois-srl** system in either  *interactive* or *annotator* mode.
+#### Interactive mode
+In *interactive mode* the user can input a single piece of text and get back the feedback from both 
+the **Nom**inal or **Verb**al SRL systems in plain text. 
+
+To run the system in *interactive mode* see the class `edu.illinois.cs.cogcomp.srl.SemanticRoleLabeler`
+or simply execute the  `run-interactive` script: 
+
+For linux:
+```
+scripts/run-interactive.sh
+```
+
+For windows:
+```
+cd scripts
+run-interactive-win.bat
+```
+
+#### As an `Annotator` component
+**illinois-srl** can also be used programmatically through the `SemanticRoleLabeler` class which implements CogComp's
+[Annotator interface](http://cogcomp.cs.illinois.edu/software/doc/illinois-core-utilities/apidocs/edu/illinois/cs/cogcomp/core/datastructures/textannotation/Annotator.html).
+
+The main method is `getView(TextAnnotation)` inside `SemanticRoleLabeler`. This will add a new 
+[`PredicateArgumentView`](http://cogcomp.cs.illinois.edu/software/doc/illinois-core-utilities/apidocs/edu/illinois/cs/cogcomp/core/datastructures/textannotation/PredicateArgumentView.html)
+for either **Nom**inal or **Verb**al SRL. 
+
+### Training
+To train the SRL system you will require access to the [Propbank](https://verbs.colorado.edu/~mpalmer/projects/ace.html)
+or [Nombank](http://nlp.cs.nyu.edu/meyers/NomBank.html) corpora. You need to set pointers to these in the 
+`config/srl-config.properties` file.
+(To train the system with a non-Prop/Nombank corpus, you need to extend 
+[`AbstractSRLAnnotationReader`](http://cogcomp.cs.illinois.edu/software/doc/illinois-core-utilities/apidocs/edu/illinois/cs/cogcomp/nlp/corpusreaders/AbstractSRLAnnotationReader.html))
+
+To perform the whole training/testing suite, run the `Main` class with parameters `<config-file> expt Verb|Nom true`.
+This will:
+
+1. Read and cache the datasets (train/test)
+2. Annotate each `TextAnnotation` with the required views
+   (here you can set the `useCurator` flag to false to use the CogComp's standalone NLP pipeline) 
+3. Pre-extract and cache the features for the classifiers
+4. Train the classifiers
+5. Evaluate on the (cached) test corpus
+
+**IMPORTANT** After training, make sure you comment-out the pre-trained SRL model dependencies inside 
+`pom.xml` (lines 27-38). 

srl/config/learner.properties

@@ -0,0 +1,20 @@
+# Available learning models: {L2LossSSVM, StructuredPerceptron}
+LEARNING_MODEL = L2LossSSVM
+
+# Available solver types: {DCDSolver, ParallelDCDSolver, DEMIParallelDCDSolver}
+L2_LOSS_SSVM_SOLVER_TYPE = ParallelDCDSolver
+
+NUMBER_OF_THREADS = 8
+
+# Regularization parameter
+C_FOR_STRUCTURE = 1.0
+
+# Mini-batch for 'warm' start
+TRAINMINI = true
+TRAINMINI_SIZE = 10000
+
+# Suppress optimatility check
+CHECK_INFERENCE_OPT = false
+
+# Number of training rounds
+MAX_NUM_ITER = 100

srl/config/pipeline.properties

@@ -0,0 +1,17 @@
+## Flags for whether to use different annotators
+usePos = true
+useLemma = true
+useShallowParse = true
+useNerConll = true
+useNerOntonotes = false
+useStanfordParse = true
+useStanfordDep = true
+useSrlVerb = false
+useSrlNom = false
+
+## Flags for the Stanford parser (for pre-processing)
+# Max time per sentence (in milliseconds)
+stanfordMaxTimePerSentence = 1000
+
+# Max sentence lenght (will throw exception if larger)
+stanfordParseMaxSentenceLength = 80

srl/config/srl-config.properties

@@ -0,0 +1,39 @@
+## Illinois SRL Configuration##
+
+# Whether to use the Illinois Curator to get the required annotations for training/testing
+# If set to false, Illinois NLP pipeline will be used
+UseCurator = false
+
+# The configuration of the Illinois NLP pipeline
+PipelineConfig = config/pipeline.properties
+
+# The parser used to extract constituents and syntactic features
+# Options are: Charniak, Berkeley, Stanford
+# NB: Only Stanford can be used in standalone mode.
+DefaultParser = Stanford
+
+# The configuration for the Structured learner
+LearnerConfig = config/learner.properties
+
+# Num of threads for feat. ext.
+NumFeatExtThreads = 10
+
+# The ILP solver to use for the joint inference
+# Options are: Gurobi, OJAlgo
+ILPSolver = OJAlgo
+
+### Training corpora directories ###
+# This is the directory of the merged (mrg) WSJ files
+PennTreebankHome = /shared/corpora/corporaWeb/treebanks/eng/pennTreebank/treebank-3/parsed/mrg/wsj/
+PropbankHome = /shared/corpora/corporaWeb/treebanks/eng/propbank_1/data
+NombankHome = /shared/corpora/corporaWeb/treebanks/eng/nombank/
+
+# The directory of the sentence and pre-extracted features database (~5G of space required)
+# Not used during test/working with pre-trained models
+CacheDirectory = cache
+
+ModelsDirectory = models
+
+# Directory to output gold and predicted files for manual comparison
+# Comment out for no output
+OutputDirectory = srl-out

srl/curator-release/README.txt

@@ -0,0 +1,20 @@
+1. Run prepare-curator-release.sh in the root directory
+
+2. Run 'ant dist' in both the verb and nom directories here.
+
+3. Copy all the jars from the lib directory into the lib directory
+   within $CURATOR_HOME/dist.
+
+4. Copy illinois-verb-srl/dist/illinois-verb-srl-server.jar and
+   illinois-nom-srl/dist/illinois-nom-srl-server.jar into
+   $CURATOR_HOME/dist/components.
+
+5. Copy config/srl-config.properties into
+   $CURATOR_HOME/dist/configs.
+
+6. Copy illinois-verb-srl/illinois-verb-srl-server.sh and
+   illinois-nom-srl/illinois-nom-srl-server.sh to the bin directory
+   within $CURATOR_HOME/dist/bin
+
+
+Now the SRL servers are ready to be launched.

srl/curator-release/config/jwnl_properties.xml

@@ -0,0 +1,45 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<jwnl_properties language="en">
+	<version publisher="Princeton" number="3.0" language="en"/>
+	<dictionary class="net.didion.jwnl.dictionary.FileBackedDictionary">
+		<param name="morphological_processor" value="net.didion.jwnl.dictionary.morph.DefaultMorphologicalProcessor">
+			<param name="operations">
+				<param value="net.didion.jwnl.dictionary.morph.LookupExceptionsOperation"/>
+				<param value="net.didion.jwnl.dictionary.morph.DetachSuffixesOperation">
+					<param name="noun" value="|s=|ses=s|xes=x|zes=z|ches=ch|shes=sh|men=man|ies=y|"/>
+					<param name="verb" value="|s=|ies=y|es=e|es=|ed=e|ed=|ing=e|ing=|"/>
+					<param name="adjective" value="|er=|est=|er=e|est=e|"/>
+                    <param name="operations">
+                        <param value="net.didion.jwnl.dictionary.morph.LookupIndexWordOperation"/>
+                        <param value="net.didion.jwnl.dictionary.morph.LookupExceptionsOperation"/>
+                    </param>
+				</param>
+				<param value="net.didion.jwnl.dictionary.morph.TokenizerOperation">
+					<param name="delimiters">
+						<param value=" "/>
+						<param value="-"/>
+					</param>
+					<param name="token_operations">
+                        <param value="net.didion.jwnl.dictionary.morph.LookupIndexWordOperation"/>
+						<param value="net.didion.jwnl.dictionary.morph.LookupExceptionsOperation"/>
+						<param value="net.didion.jwnl.dictionary.morph.DetachSuffixesOperation">
+							<param name="noun" value="|s=|ses=s|xes=x|zes=z|ches=ch|shes=sh|men=man|ies=y|"/>
+							<param name="verb" value="|s=|ies=y|es=e|es=|ed=e|ed=|ing=e|ing=|"/>
+							<param name="adjective" value="|er=|est=|er=e|est=e|"/>
+                            <param name="operations">
+                                <param value="net.didion.jwnl.dictionary.morph.LookupIndexWordOperation"/>
+                                <param value="net.didion.jwnl.dictionary.morph.LookupExceptionsOperation"/>
+                            </param>
+						</param>
+					</param>
+				</param>
+			</param>
+		</param>
+		<param name="dictionary_element_factory" value="net.didion.jwnl.princeton.data.PrincetonWN17FileDictionaryElementFactory"/>
+		<param name="file_manager" value="net.didion.jwnl.dictionary.file_manager.FileManagerImpl">
+			<param name="file_type" value="net.didion.jwnl.princeton.file.PrincetonRandomAccessDictionaryFile"></param>
+			<param name="dictionary_path" value="/shared/grandpa/opt/dict"/>
+		</param>
+	</dictionary>
+	<resource class="JWNLResource"></resource>
+</jwnl_properties>

srl/curator-release/config/srl-config.properties

@@ -0,0 +1,10 @@
+# For a description of the arguments, see the documents
+DefaultParser = Charniak
+
+WordNetConfig=jwnl_properties.xml
+
+
+StandardIdentifierClassifierPipeline = true
+
+
+

srl/curator-release/illinois-nom-srl/README.dist

@@ -0,0 +1,48 @@
+Illinois Nominal SRL Server
+======================
+
+Description
+-----------
+
+This package wraps the Illinois Nominal Semantic Role Labeler as a
+Thrift server using the Curator's Parser interface. It is designed to
+be integrated into the Curator system. Parsers return a forests; in
+this case, consisting of one tree per predicate, with the predicate as
+the root and its arguments as the children.
+
+
+Semantics
+----------
+
+Semantic Roles are represented using a forest consisting of two level
+trees. Each predicate in the input is associated with a tree whose
+root corresponds to the predicate. The children of this node denote
+the labeled arguments of the predicate.
+
+Usage
+-----
+
+>     $ bin/illinois-verb-srl-server.sh --help
+>     usage: java edu.illinois.cs.cogcomp.annotation.server.IllinoisNomSRLServer
+>            [-c <CONFIG>] [-h] [-p <PORT>] [-t <THREADS>]
+>      -c,--config <CONFIG>     configuration file
+>      -h,--help                print this message
+>      -p,--port <PORT>         port to open server on
+>      -t,--threads <THREADS>   number of threads to run
+
+Installation
+------------
+
+The Illinois Nominal SRL Server depends on:
+
+* Java >= 1.5
+* The Illinois SRL system
+* Illinois SRL model jar
+* Curator interfaces (`curator-interfaces.jar`)
+
+
+Configuration
+-------------
+
+See the documentation for the Illinois SRL system for configuration
+options.