n-grams

A pure pythonic n-gram program that allows you to generate a random sentence from a training set, compute perplexity of a test set on a training set, and perform a perplexity-based multi-class classification. This program requires no outside packages, no installation, and runs on all Python 2.7 and above.

Usage

The usage of this n-gram interface is:

ngrams.py [-h] [-n N] -sent training_set
          [-h] [-n N] [-sent] [-t T] [-ls [α] | -gts] [-p] [-c [CATEGORY_NUM] output_file] training_set test_set

Running the command:

./ngrams.py -h

or

python ngrams.py -h

will provide a help message, with some explanation for each option.

Generate Random Sentence

You can generate a random sentence by inputting the -sent option, and a text file. This will generate a random sentence based on an unsmoothed n-gram model. Note: you can insert an 'n' by inserting the -n flag followed by the desired n; if no n is inserted, n is set to 2 (bigrams).

Example:

python ngrams.py -sent -n 4 review.train
It is one of chicago 's best recently renovated to bring it up .

Perplexity

You can find the perplexity of two pieces of text using the -p option, and inserting the two text files. You can also choose which type of smoothing you would like to use: Laplace (Additive) or Good Turing smoothing. If using Laplacian smoothing, you can set the smoothing parameter by adding a float value after the -ls option. Note: Good Turing smoothing has only been implemented for unigrams and bigrams. Also, Laplacian smoothing is used by default. In addition, this program parses texts slightly differently if it knows that it will be classified.

Example:

./ngrams.py -ls .2 -p kjbible.train kjbible.test
Perplexity: 216.38605101456963

The perplexity will slightly depend on the Python version, as the math module was updated in Python 3.x.

Multi-Class Classification

You can classify text a pieces of text by providing a training set and the test set you wish to classify. The examples provided in the test set will have their perplexities compared to every class in the training set in order to classify each example.

Example:

./ngrams.py -c classifications.txt reviews.train reviews.test

Putting it All Together

You can do all of these operations at once. Note: the train and test sets must be at the end of every command, in that order.

Example:

./ngrams.py -n 2 -sent -ls .2 -p -c 1 classes.txt reviews.train reviews.test
When I complained I asked for us to be transferred to the Sheraton which they arranged for us .
Perplexity: 218.65148036450822