Added Snowball stemmer to class Normalization.java #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
lorikdumani
commented
Feb 23, 2018
lorikdumani
commented
- Also added a test case
* the similarity threshold is now only defined in the so-posthistory-extractor project
* select "rightmost" minimal hash value in window * remove duplicate hash values (considering position) * update methods fingerprintList and completeFingerprintList * update test case
* robust winnowing as described in Schleimer03 * window size is now always twice the nGram size (as in Schleimer03) * update test case
* if input is too short, throw InputTooShortException
* now checking size of input strings for ngram and shingle metrics
* implement default metrics and variants using longestCommonSubsequence and optimalAlignment * made required edit base metrics public
* re-added metrics combining winnowing with edit-based metrics
* add file encoding * update some versions * remove unnecessary dependencies * add comments
* InputTooShortException in now thrown there and not in metrics using those methods * update test cases * add new test case checking if InputTooShortException in thrown
* won't implement this
* moved length check to nGramList and shingleList * refactorings
Reason: Values may be larger than 1.0:
str1 = "abc"
str2 = "abc"
2-Grams: {"ab", "bc"}
Intersection: {"ab", "bc"}
Dice variant: 2*2 / 2 = 2
* refactored test cases * removed dice variant based metrics
* issue when one string was empty * now correctly ordering strings * add test case
* fixed bug in levenshtein based metrics
* revise test cases
* Kondrak05-based metrics now also throw InputTooShortException if input length is shorter than nGram size
* fix bug in optimal alignment * revise Kondrak05-based metrics * revise test cases
* rename method Base.equals to Base.equal to avoid confusion with Object.equals
* move equality-based metrics to package "equal" * rename method Base.equals to Base.equal to avoid confusion with Object.equals
* move util methods to util project
* update default values after evaluation
* now calling listToSet instead of nGramSet (copy&paste error)
* fix bug in token-based similarity
* Implement a method to normalize text blocks by a stemmer
* Also added a test case
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.