code/preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocess corpora\n",
    "##### See [preprocessing.py](https://github.com/devmount/GermanWordEmbeddings/blob/master/preprocessing.py) from [GermanWordEmbeddings](https://devmount.github.io/GermanWordEmbeddings/)\n",
    "\n",
    "The following code gives an example of how to use the preprocessing script to filter and transform a given corpus. You need [gensim](https://radimrehurek.com/gensim/install.html) and [NLTK](http://www.nltk.org/install.html) for this script to work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### General usage\n",
    "The usage of the script can be seen with the default `-h` or `--help` flag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: preprocessing.py [-h] [-p] [-s] [-u] [-b] raw target\n",
      "\n",
      "Script for preprocessing public corpora\n",
      "\n",
      "positional arguments:\n",
      "  raw                source file with raw data for corpus creation\n",
      "  target             target file name to store corpus in\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help         show this help message and exit\n",
      "  -p, --punctuation  remove punctuation tokens\n",
      "  -s, --stopwords    remove stop word tokens\n",
      "  -u, --umlauts      replace german umlauts with their respective digraphs\n",
      "  -b, --bigram       detect and process common bigram phrases\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python preprocessing.py --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Raw corpus example\n",
    "\n",
    "To show the results of the preprocessing script, a corpus with 1k sentences from German news data is used. Here's the first 5 sentences of this unprocessed corpus:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zwei Männer fuhren unter Drogeneinfluss.\n",
      "Das war im Juli dieses Jahres.\n",
      "Krimi-Serie: Im Tatort verteilt der Entführer 500.000 Euro am Alex.\n",
      "Er sagte aber nicht, für wie lange.\n",
      "Insgesamt investierte die Baugenossenschaft dafür 2,4 Millionen Euro.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "head -n 5 corpus/corpus.raw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove punctuation\n",
    "\n",
    "Punctuation is removed with the `-p` or `--punctuation` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:preprocessing 1000 sentences\n",
      "INFO:root:preprocessing of 1000 sentences finished!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python preprocessing.py -p corpus/corpus.raw corpus/corpus.nopunct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zwei Männer fuhren unter Drogeneinfluss\n",
      "Das war im Juli dieses Jahres\n",
      "Krimi-Serie Im Tatort verteilt der Entführer 500000 Euro am Alex\n",
      "Er sagte aber nicht für wie lange\n",
      "Insgesamt investierte die Baugenossenschaft dafür 2,4 Millionen Euro\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "head -n 5 corpus/corpus.nopunct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You see that the dot in the 500k Euro was removed, but the comma in 2,4M Euro wasn't. So numbers were not falsified. The dash between compund words (*Krimi-Serie*) was not removed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove stopwords\n",
    "\n",
    "Stopwords are removed with the `-s` or `--stopwords` flag. Therefore a list of German stopwords from NLTK is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:preprocessing 1000 sentences\n",
      "INFO:root:preprocessing of 1000 sentences finished!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python preprocessing.py -ps corpus/corpus.raw corpus/corpus.nostop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zwei Männer fuhren Drogeneinfluss\n",
      "Das Juli Jahres\n",
      "Krimi-Serie Im Tatort verteilt Entführer 500000 Euro Alex\n",
      "Er sagte lange\n",
      "Insgesamt investierte Baugenossenschaft dafür 2,4 Millionen Euro\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "head -n 5 corpus/corpus.nostop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Words like *im*, *aber*, *nicht* and *wie* are removed from the example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Transform umlauts\n",
    "\n",
    "German umlauts are transformed with the `-u` or `--umlauts` flag, so that they become their respective digraph representation, e.g. *ä* -> *ae* and *ß* -> *ss*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:preprocessing 1000 sentences\n",
      "INFO:root:preprocessing of 1000 sentences finished!\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python preprocessing.py -psu corpus/corpus.raw corpus/corpus.nouml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zwei Maenner fuhren Drogeneinfluss\n",
      "Das Juli Jahres\n",
      "Krimi-Serie Im Tatort verteilt Entfuehrer 500000 Euro Alex\n",
      "Er sagte lange\n",
      "Insgesamt investierte Baugenossenschaft dafuer 2,4 Millionen Euro\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "head -n 5 corpus/corpus.nouml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bigram phrases\n",
    "\n",
    "The corpus is transformed to bigram phrases with the `-b` or `--bigram` flag. In the example corpus the words *Millionen* and *Euro* are transformed to one token *Millionen_Euro*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:preprocessing 1000 sentences\n",
      "INFO:root:preprocessing of 1000 sentences finished!\n",
      "INFO:root:train bigram phrase detector\n",
      "INFO:gensim.models.phrases:collecting all words and their counts\n",
      "INFO:gensim.models.phrases:PROGRESS: at sentence #0, processed 0 words and 0 word types\n",
      "INFO:gensim.models.phrases:collected 13779 word types from a corpus of 8044 words (unigram + bigrams) and 995 sentences\n",
      "INFO:gensim.models.phrases:merging 13779 counts into Phrases<0 vocab, min_count=5, threshold=10.0, max_vocab_size=40000000>\n",
      "INFO:gensim.models.phrases:merged Phrases<13779 vocab, min_count=5, threshold=10.0, max_vocab_size=40000000>\n",
      "INFO:root:transform corpus to bigram phrases\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "python preprocessing.py -psub corpus/corpus.raw corpus/corpus.psu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Zwei Maenner fuhren Drogeneinfluss\n",
      "Das Juli Jahres\n",
      "Krimi-Serie Im Tatort verteilt Entfuehrer 500000 Euro Alex\n",
      "Er sagte lange\n",
      "Insgesamt investierte Baugenossenschaft dafuer 2,4 Millionen_Euro\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "head -n 5 corpus/corpus.psu.bigram"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}