ensemble-methods-notebooks-master/Ch3.4-case-study-sentiment-analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_This notebook contains code and comments from Section 3.4 of the book [Ensemble Methods for Machine Learning](https://www.manning.com/books/ensemble-methods-for-machine-learning). Please see the book for additional details on this topic. This notebook and code are released under the [MIT license](https://github.com/gkunapuli/ensemble-methods-notebooks/blob/master/LICENSE)._\n",
    "\n",
    "---\n",
    "\n",
    "## 3.4 Case Study: Sentiment Analysis\n",
    "\n",
    "Sentiment analysis is a natural language processing (NLP) task for identifying the the polarity of opinion as positive, neutral or negative. This case study explores a supervised sentiment analysis task for movie reviews. The data set we will use is the [Large Movie Review Dataset](https://ai.stanford.edu/~amaas/data/sentiment/), which was originally collected and curated for a 2011 paper on sentiment analysis: \n",
    "\n",
    "Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. (2011). _Learning Word Vectors for Sentiment Analysis_. The 49th Annual Meeting of the Association for Computational Linguistics (ACL 2011)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data directory relative to this notebook\n",
    "sentiment_data_directory = './data/ch03/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.1 Pre-processing\n",
    "This data set has already been pre-processed by [count vectorization](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html) to generate [bag-of-word features](https://en.wikipedia.org/wiki/Bag-of-words_model). These pre-processed term-document count features, our data set, can be found in ``./data/ch03/train/labeledBow.feat`` and ``./data/ch03/test/labeledBow.feat``.\n",
    "\n",
    "#### Stop-word Removal\n",
    "This step aims to remove common words such as “the”, “is”, “a”, “an”. Stop word removal can reduce the dimensionality of the data, (to make processing faster), and can improve classification performance. This is because words like “the” are often not really informative for information retrieval and text-mining tasks.\n",
    "\n",
    "**Listing 3.11**: Drop stop words from the vocabulary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "import time\n",
    "import numpy as np\n",
    "\n",
    "def prune_vocabulary(data_path, max_features=5000):\n",
    "    start_time = time.time()\n",
    "    with open('{0}/imdb.vocab'.format(data_path), 'r', encoding='utf8') as vocab_file:\n",
    "        vocabulary = vocab_file.read().splitlines()\n",
    "    print('Vocabulary load time = {0} seconds.'.format(time.time() - start_time))\n",
    "\n",
    "    # Convert the list of stopwords to a set for faster processing\n",
    "    # nltk.download('stopwords')\n",
    "    stopwords = set(nltk.corpus.stopwords.words(\"english\"))\n",
    "\n",
    "    # Keep only those vocabulary words that are NOT stopwords\n",
    "    to_keep = [True if word not in stopwords else False for word in vocabulary]\n",
    "    feature_ind = np.where(to_keep)[0]\n",
    "\n",
    "    return feature_ind[:max_features]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Vocabulary load time = 0.016955852508544922 seconds.\n"
     ]
    }
   ],
   "source": [
    "features = prune_vocabulary(sentiment_data_directory, max_features=5000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TF-IDF Transformation\n",
    "Our second pre-processing step converts the count features to [tf-idf features](https://en.wikipedia.org/wiki/Tf%E2%80%93idf). These features represent the term frequency-inverse document frequency, a statistic that weights each feature in a document (in our case, a single review) relative to how often it appears in that document as well as how often it appears in the entire corpus (in our case, all the reviews). \n",
    "\n",
    "We can use scikit-learn’s pre-processing toolbox to convert our count features to tf-idf features using the [TfidfTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html).\n",
    "\n",
    "**Listing 3.12**: Extract tf-idf features and save the data set (**NOTE: _generates a 41 MB file_**)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h5py\n",
    "from sklearn.datasets import load_svmlight_files\n",
    "from scipy.sparse import csr_matrix as sp\n",
    "from sklearn.feature_extraction.text import TfidfTransformer\n",
    "\n",
    "def preprocess_and_save(data_path, feature_ind):\n",
    "    data_files = ['{0}/{1}/labeledBow.feat'.format(data_path, data_set) \n",
    "                  for data_set in ['train', 'test']]\n",
    "    [Xtrn, ytrn, Xtst, ytst] = load_svmlight_files(data_files)\n",
    "    n_features = len(feature_ind)\n",
    "\n",
    "    ytrn[ytrn <= 5], ytst[ytst <= 5] = 0, 0\n",
    "    ytrn[ytrn > 5], ytst[ytst > 5] = 1, 1\n",
    "\n",
    "    # Transform the bag-of-words\n",
    "    tfidf = TfidfTransformer()\n",
    "    Xtrn = tfidf.fit_transform(Xtrn[:, feature_ind])\n",
    "    Xtst = tfidf.transform(Xtst[:, feature_ind])\n",
    "      \n",
    "    # Save the data in HDF5 format with sparse matrix representation\n",
    "    with h5py.File('{0}/imdb-{1}k.h5'.format(data_path, \n",
    "                            round(n_features/1000)), 'w') as db:\n",
    "        db.create_dataset('Xtrn', \n",
    "                          data=sp.todense(Xtrn), compression='gzip')\n",
    "        db.create_dataset('ytrn', data=ytrn, compression='gzip')\n",
    "        db.create_dataset('Xtst', \n",
    "                          data=sp.todense(Xtst), compression='gzip')\n",
    "        db.create_dataset('ytst', data=ytst, compression='gzip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocess_and_save(sentiment_data_directory, features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "### 3.4.2\tDimensionality Reduction\n",
    "\n",
    "We adopt the popular dimensionality reduction approach of [principal components analysis](https://en.wikipedia.org/wiki/Principal_component_analysis) (PCA), which aims to compress and embed the data into a lower-dimensional representation while while preserving as much of the information as possible. \n",
    "\n",
    "To avoid loading the entire data set into memory and to process the data more efficiently, we perform [Incremental PCA](https://scikit-learn.org/stable/auto_examples/decomposition/plot_incremental_pca.html) instead.\n",
    "\n",
    "**Listing 3.13**: Perform dimensionality reduction using Incremental PCA (**NOTE: _generates a 187 MB file_**)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.decomposition import IncrementalPCA\n",
    "\n",
    "def transform_sentiment_data(data_path, n_features=5000, n_components=500):\n",
    "    db = h5py.File('{0}/imdb-{1}k.h5'.format(data_path, round(n_features/1000)), 'r')\n",
    "\n",
    "    pca = IncrementalPCA(n_components=n_components)\n",
    "    chunk_size = 1000\n",
    "    n_samples = db['Xtrn'].shape[0] \n",
    "    for i in range(0, n_samples // chunk_size):\n",
    "        pca.partial_fit(db['Xtrn'][i*chunk_size:(i+1) * chunk_size])\n",
    "\n",
    "    Xtrn = pca.transform(db['Xtrn'])\n",
    "    Xtst = pca.transform(db['Xtst'])\n",
    "\n",
    "    with h5py.File('{0}/imdb-{1}k-pca{2}.h5'.format(data_path,\n",
    "                                                    round(n_features/1000), n_components), 'w') as db2:\n",
    "        db2.create_dataset('Xtrn', data=Xtrn, compression='gzip')\n",
    "        db2.create_dataset('ytrn', data=db['ytrn'], compression='gzip')\n",
    "        db2.create_dataset('Xtst', data=Xtst, compression='gzip')\n",
    "        db2.create_dataset('ytst', data=db['ytst'], compression='gzip')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "transform_sentiment_data(sentiment_data_directory, n_features=5000, n_components=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "###  3.4.3\tStacking classifiers\n",
    "\n",
    "Our goal now is to train a heterogeneous ensemble with meta-learning. Specifically, we will use ensemble several base estimators by blending them. Blending is a variant of stacking, where, instead of using cross validation, we use a single validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_sentiment_data(data_path,n_features=5000, n_components=500):\n",
    "\n",
    "    with h5py.File('{0}/imdb-{1}k-pca{2}.h5'.format(data_path, \n",
    "                                                    round(n_features/1000), n_components), 'r') as db:\n",
    "        Xtrn = np.array(db.get('Xtrn'))\n",
    "        ytrn = np.array(db.get('ytrn'))\n",
    "        Xtst = np.array(db.get('Xtst'))\n",
    "        ytst = np.array(db.get('ytst'))\n",
    "\n",
    "    return Xtrn, ytrn, Xtst, ytst"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "Xtrn, ytrn, Xtst, ytst = load_sentiment_data('./data/Ch03/', n_features=5000, n_components=500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we use five base estimators: [``RandomForestClassifier``](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) with 100 randomized decision trees, [``ExtraTreesClassifier``] (https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html#sklearn.ensemble.ExtraTreesClassifier) with 100 extremely randomized trees, [``Logistic Regression``](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html), Bernoulli [naïve Bayes](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.naive_bayes) (``BernoulliNB``) and a linear SVM trained with stochastic gradient descent ([``SGDClassifier``](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html))."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier, ExtraTreesClassifier\n",
    "from sklearn.linear_model import LogisticRegression, SGDClassifier\n",
    "from sklearn.naive_bayes import BernoulliNB\n",
    "\n",
    "estimators = [('rf', RandomForestClassifier(n_estimators=100, n_jobs=-1)),\n",
    "              ('xt', ExtraTreesClassifier(n_estimators=100, n_jobs=-1)),\n",
    "              ('lr', LogisticRegression(C=0.01, solver='lbfgs')),\n",
    "              ('bnb', BernoulliNB()),\n",
    "              ('svm', SGDClassifier(loss='hinge', penalty='l2', alpha=0.01,\n",
    "                                    n_jobs=-1, max_iter=10, tol=None))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To blend these base-estimators into a heterogeneous ensemble with meta-learning, we use the following procedure:\n",
    "1.\tSplit the training data into a training set ``(Xtrn, ytrn)`` with 80% of the data and a validation set ``(Xval, yval)``, with the remaining 20% of the data\n",
    "2.\tTrain the each of the level-1 estimators on the training set, ``(Xtrn, ytrn)``\n",
    "3.\tGenerate meta-features ``Xmeta`` with the trained estimators using ``Xval``; \n",
    "4.\tAugment the validation data with the meta-features: ``[Xval, Xmeta]``; this augmented validation set will have 500 original features + 5 meta-features\n",
    "5.\tTrain the level-2 estimator with the augmented validation set ``([Xval, Xmeta], yval)``\n",
    "\n",
    "This leaves one final decision: the choice of the level-2 estimator. Previously, we used simple linear classifiers. For this classification task, we utilize a neural network. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.neural_network import MLPClassifier\n",
    "meta_estimator = MLPClassifier(hidden_layer_sizes=(128, 64, 32),\n",
    "                               alpha=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Listing 3.14**: Blending models with a validation set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "def blend_models(level1_estimators, level2_estimator, \n",
    "                 X, y , use_probabilities=False):    \n",
    "    Xtrn, Xval, ytrn, yval = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "    n_estimators = len(level1_estimators)\n",
    "    n_samples = len(yval)\n",
    "    Xmeta = np.zeros((n_samples, n_estimators))\n",
    "    for i, (model, estimator) in enumerate(level1_estimators):\n",
    "        estimator.fit(Xtrn, ytrn)\n",
    "        Xmeta[:, i] = estimator.predict(Xval)\n",
    "\n",
    "    Xmeta = np.hstack([Xval, Xmeta])\n",
    "    \n",
    "    level2_estimator.fit(Xmeta, yval)\n",
    "\n",
    "    final_model = {'level-1': level1_estimators, \n",
    "                   'level-2': level2_estimator, \n",
    "                   'use-proba': use_probabilities}\n",
    "\n",
    "    return final_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Listing 3.2** for individual base estimator predictions and **Listing 3.9** for meta-estimator predictions are combined here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_stacking(X, stacked_model):\n",
    "    # Get level-1 predictions\n",
    "    level1_estimators = stacked_model['level-1']\n",
    "    n_samples, n_estimators = X.shape[0], len(level1_estimators)\n",
    "    use_probabilities = stacked_model['use-proba']\n",
    "\n",
    "    Xmeta = np.zeros((n_samples, n_estimators))  # Initialize meta-features\n",
    "    for i, (model, estimator) in enumerate(level1_estimators):\n",
    "        if use_probabilities:\n",
    "            Xmeta[:, i] = estimator.predict_proba(X)[:, 1]\n",
    "        else:\n",
    "            Xmeta[:, i] = estimator.predict(X)\n",
    "\n",
    "    level2_estimator = stacked_model['level-2']\n",
    "    Xmeta = np.hstack([X, Xmeta])\n",
    "\n",
    "    y = level2_estimator.predict(Xmeta)\n",
    "\n",
    "    return y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a simple blending model with this code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacked_model = blend_models(estimators, meta_estimator, Xtrn, ytrn)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "7.8359999999999985\n",
      "17.196\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "ypred = predict_stacking(Xtrn, stacked_model)\n",
    "trn_err = (1 - accuracy_score(ytrn, ypred)) * 100\n",
    "print(trn_err)\n",
    "\n",
    "ypred = predict_stacking(Xtst, stacked_model)\n",
    "tst_err = (1 - accuracy_score(ytst, ypred)) * 100\n",
    "print(tst_err)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "### Overall Performance of the blended model\n",
    "We finally visualize and comparing the performance of each individual base classifier with the meta-classifier ensemble. Stacking/blending improves classification performance by ensembling diverse base classifiers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "rf: training error = 0.00%, test error = 22.61%, running time = 10.22 seconds.\n",
      "xt: training error = 0.00%, test error = 30.00%, running time = 4.44 seconds.\n",
      "lr: training error = 17.65%, test error = 18.03%, running time = 0.13 seconds.\n",
      "bnb: training error = 21.66%, test error = 22.84%, running time = 0.35 seconds.\n",
      "svm: training error = 22.84%, test error = 23.05%, running time = 0.41 seconds.\n"
     ]
    }
   ],
   "source": [
    "trn_errors, tst_errors = np.zeros((len(estimators) + 1, )), np.zeros((len(estimators) + 1, ))\n",
    "for i, (method, estimator) in enumerate(estimators):\n",
    "    start_time = time.clock()\n",
    "    estimator.fit(Xtrn, ytrn)\n",
    "    run_time = time.clock() - start_time\n",
    "\n",
    "    ypred = estimator.predict(Xtrn)\n",
    "    trn_errors[i] = (1 - accuracy_score(ytrn, ypred)) * 100\n",
    "    ypred = estimator.predict(Xtst)\n",
    "    tst_errors[i] = (1 - accuracy_score(ytst, ypred)) * 100\n",
    "\n",
    "    print('{0}: training error = {1:4.2f}%, test error = {2:4.2f}%, running time = {3:4.2f} seconds.'\n",
    "          .format(method, trn_errors[i], tst_errors[i], run_time))\n",
    "    \n",
    "trn_errors[-1] = trn_err\n",
    "tst_errors[-1] = tst_err"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "def autolabel(ax, rects):\n",
    "    for rect in rects:\n",
    "        height = np.round(rect.get_height(), 1)\n",
    "        ax.annotate('{}'.format(height),\n",
    "                    xy=(rect.get_x() + rect.get_width() / 2, height),\n",
    "                    xytext=(0, 3),  # 3 points vertical offset\n",
    "                    textcoords=\"offset points\",\n",
    "                    ha='center', va='bottom')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAAEXCAYAAAD4LtBgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXgUVdb48e8hYV+H1UCQRQTCZkA2NwyjLIOKIiggjgvwIr4o7o7j8htGZxz01VERHQdfBdwAfVFBBBSQFlQQQQKGTRzIyKYiI6soJJzfH1UdOkl30gld3Z30+TxPnnRVV906t6u7b9etW6dEVTHGGGPiTYVYB2CMMcYEYw2UMcaYuGQNlDHGmLhkDZQxxpi4ZA2UMcaYuGQNlDHGmLhkDZQJm4hUEZFVIrJORDaIyJ/d+XVFZJGIbHX//ybE+v1FZIuIfCMi90U3emNMWSNl4Tqo+vXra/PmzWMdRiE///wz1apVi3UYUXPkyBGqVKlCUlISqsrmzZtp2rQp+/fvJzk5mdNOO43vvvuOnJwcUlNT862rqmRlZdG6dWsqVqzI5s2badGiBVWrVo1RbYqXaPvX6lu+xWt916xZ86OqNgj6pKrG/d/ZZ5+t8Wjp0qWxDiGqAut75MgR7dy5s65cuVJbt26tu3fvVlXV3bt3a+vWrQut+9lnn2nfvn3zph999FF99NFHPY/5VCTy/k0EVt/4AKzWEN/91sVnSiQ3N5f09HQaNmxInz596NGjB99//z0pKSkApKSk8MMPPxRab9euXTRt2jRvOjU1lV27dkUtbmNM2WMNlCmRpKQkMjMz2blzJ6tWrSIrKyus9TRIV7KIRDo8Y0w5Yg2UKZU6deqQkZHBwoULadSoEXv27AFgz549NGzYsNDyqamp7NixI296586dNG7cOGrxGmPKnuRYB2DKjv3797N//37q1KnD0aNHWbx4MX/4wx8YOHAg06dP57777mP69Olcfvnlhdbt1q0bW7duZfv27TRp0oSZM2fyxhtvxKAWprw7fvw4O3fu5Jdffilyudq1a7Np06YoRRV7sa5vlSpVSE1NpWLFimGvYw2UCdu+ffvo3bs3ubm5nDhxgquvvppLL72Uc845h6uvvpqXXnqJ008/nbfeeguA3bt3M3r0aObPn09ycjKTJ0+mX79+5ObmMnLkSNq3bx/jGpnyaOfOndSsWZPmzZsX2Y186NAhatasGcXIYiuW9VVV9u3bx86dO2nRokXY61kDZcJ2xhlnsHbt2kLz69Wrx5IlSwrNb9y4MfPnz8+bHjBgAAMGDPA0RmN++eWXYhsnE10iQr169di7d2+J1vPsHNSpXtRpjDGlZY1T/CnNPvFykMSvwG9V9SwgHegvIj2B+4AlqnomsMSdNsYYY/LxrIvPvQDrsDtZ0f1T4HIgw50/HfABf/AqDmNMYmt+3/sRLS974iVFPr9v3z4uuugiAL777juSkpJo0MBJlLBq1SoqVaoUct3Vq1fzyiuvMGnSpCK3ce655/LZZ5+VMPLCfD4fl19+eb7zQk888QQXX3zxKZcdCZ6mOhKRJGAN0Ap4TlX/ICL7VbVOwDI/qWqhbj4RGQOMAUhJSTk7Hkd8ZWdnE48pmLyS4Ss8Ou9U+TLmRLzMSEm0/Vte6lu7dm1atWqVN93xr8siWv5XD/QKe9lHH32UGjVqMH78+Lx5OTk5JCdH//T/r7/+SuXKlfPNW758OZMmTcob2BRMXlaHChWCToeSm5tLUlJSvnnffPMNBw4cyDevd+/ea1S1a7AyPH2VVDUXSBeROsA7ItKhBOtOAaYAdO3aVTMyMrwJ8hT4fD7iMS7P+CJfZDy/fom2f8tLfTdt2uTpaLWSlF25cmUqV67MrbfeSt26dVm7di1dunRh6NCh3H777Rw9epSqVasydepU2rRpg8/n44knnmDevHlMmDCBb7/9lm3btvHtt99y++235zV0NWrU4PDhw/h8PiZMmED9+vXJysri7LPP5rXXXkNEmD9/PnfeeSf169enS5cufP311yxcuDBffNWqVSM5OblQnbKzs/nd735H7969WbFiBU8//TRjx47Nm3733XeZPHkyCxYsQER48MEHGTp0KD6fjz//+c+kpKSQmZnJxo0b85VbpUoVOnfuHPbrF5VmXFX3i4gP6A98LyIpqrpHRFKAwnlxjDGmnPn6669ZvHgxSUlJHDx4kGXLlpGcnMzixYu5//77mT17dqF1Nm/ezNKlSzl06BBt2rTh5ptvLnQd0dq1a9mwYQONGzfmvPPO49NPP6Vr167cdNNNLFu2jBYtWjB8+PCQcS1fvpz09PS86dmzZ5OUlMSWLVuYOnUqzz//PNnZ2fmmZ8+eTWZmJuvWrePHH3+kW7du9OrlHFn6M8yUZDh5KJ41UCLSADjuNk5VgYuBx4C5wPXARPd//PbxGGNMhFx11VV5XV4HDhzg+uuvZ+vWrYgIx48fD7rOJZdckncU1rBhQ77//vtCdwro3r173rz09HSys7OpUaMGLVu2zGskhg8fzvPPPx90GxdccAHz5s3LNy87O5tmzZrRs2fPvHmB05988gnDhw8nKSmJRo0aceGFF/LFF19Qq1YtunfvHpHGCbwdxZcCLBWR9cAXwCJVnYfTMPURka1AH3faGGPKterVq+c9fuihh+jduzdZWVm89957IbNeBJ4zSkpKIicnJ6xlIjG2IDDegtNFlV9wvVPhWQOlqutVtbOqdlLVDqr6sDt/n6pepKpnuv//41UMxhgTjw4cOECTJk0AmDZtWsTLb9u2Ldu2bSM7OxuAWbNmRbT8Xr16MWvWLHJzc9m7dy/Lli2je/fuEd0GWCYJY0w5F2pYeCxT/9x7771cf/31/P3vf+e3v/1txMuvWrUqzz//PP3796d+/fp07949ZDdiwXNQDz74IF27Bh1Ul2fQoEGsWLGCs846CxHh8ccf57TTTmPz5s0RrUeZuKNu165ddfXq1bEOo5DyMuopbBNqe1DmgeKXiZFE27/lpb6bNm0iLS2t2OXKey6+w4cPU6NGDVSVcePG0bRpU/74xz/GNKZg+0ZEQg4zt9ttGGNMOfTiiy+Snp5O+/btOXDgACNHjox1SCVmXXzGGFMO3XHHHdxxxx1504cOHYphNKVjR1DGGGPikjVQxhhj4pI1UMYYY+KSNVDGGGPikg2SMMaUbyEujyj1APNiLo04ldttgDPcv1KlSpx77rmFnps2bRr33HNP3kW+AG+88Qbt2rUraS3KBGugjDEmgurVq0dmZiYAEyZMoEaNGtx9991hr+/z+ahRo0bQBgpg6NChTJ48OeT6BW9zEey2F8HE6jYgRbEuPmOM8diaNWu48MILOfvss+nXrx979uwBYNKkSbRr145OnToxbNgwsrOzeeGFF3jqqadIT09n+fLlYZXv8/no3bs311xzDR07diw0/csvv3DzzTfTsWNHOnfuzNKlSwHniOyqq67isssuo2/fvp7Vv7Tiq7k0xphyRlW59dZbmTNnDg0aNGDWrFk88MADvPzyy0ycOJHt27dTuXJl9u/fT506dRg7dmyRR12zZs3ik08+yZtesWIFkP82Fz6fL9/0k08+CcBXX33F5s2b6du3L19//XXe+uvXr6du3boevxIlZw2UMcZ46NdffyUrK4s+ffoATpdbSkoKAJ06dWLEiBFcccUVXHHFFWGVF6qLr+BtLgKnP/nkE0aNGgU4iWSbNWuW10D16dMnLhsnsAbKGGM8paq0b98+70gn0Pvvv8+yZcuYO3cujzzyCBs2bCj1duLh9hiRZuegjDHGQ5UrV2bv3r15DdTx48fZsGEDJ06cYMeOHfTu3ZvHH3+c/fv3c/jwYWrWrBnxtES9evXizTffBJw7+3777be0adMmotvwgh1BGWPKtxDDwqOVzbxChQr83//9H+PHj+fAgQPk5ORw++2307p1a6699loOHDiAqnLHHXdQp04dLrvsMoYMGcKcOXN49tlnueCCC/KVV/AcVKg75Qb67//+b0aNGkXHjh1JTk5m2rRp+W50GK/sdhunoLzcniBsdruNcq281NdutxFcPNTXbrdhjDGmXLAGyhhjTFyyBsoYU+6UhVMXiaY0+8QaKGNMuVKlShX27dtnjVQcUVX27dtHlSpVSrSejeIzxpQrqamp7Ny5k7179xa53C+//FLiL8yyLNb1rVKlCqmpqSVaxxooY0y5UrFixXwZFULx+Xx07tw5ChHFh7JYX+viM8YYE5esgTLGGBOXrIEyxhgTl6yBMsYYE5esgTLGGBOXrIEyxhgTlzxroESkqYgsFZFNIrJBRG5z508QkV0ikun+DfAqBmOMMWWXl9dB5QB3qeqXIlITWCMii9znnlLVJzzctjHGmDKuyAZKRKoAlwIXAI2Bo0AW8L6qFnnrR1XdA+xxHx8SkU1Ak0gEbYwxpvwL2UCJyATgMsAHfA78AFQBWgMT3cbrLlVdX9xGRKQ50Nkt5zzgFhG5DljtlvFTkHXGAGMAUlJS8Pl84dcqSrKzs+MyLq9keFBmPL9+ibZ/rb7lW1msb8gbForIJar6fsgVRRoCp6tqkXcSFJEawMfAX1X1bRFpBPwIKPAIkKKqI4sqw25YGCfshoXlmtW3fIvX+pbqhoXBGicRqSIitdznfwijcaoIzAZeV9W33fW+V9VcVT0BvAh0D78qxhgTGT/88AO9e/cmLS2N9u3b88wzzwDw0EMP0alTJ9LT0+nbty+7d+8Ouv7ChQtp06YNrVq1YuLEidEMPWGEPYpPREYDHwDvi8ijYSwvwEvAJlX9e8D8lIDFBuGc0zLGmKhKSkriySefZNOmTaxcuZLnnnuOjRs3cs8997B+/XoyMzO59NJLefjhhwutm5uby7hx41iwYAEbN25kxowZbNy4MQa1KN9CNlAiclmBWRer6oWqegFwSRhlnwf8HvhtgSHlj4vIVyKyHugN3FHa4I0xkRPqiOKee+6hbdu2dOrUiUGDBrF///6g6z/11FO0b9+eDh06MHz4cH755ZeIxbZjx46Ix1avXj26dOkCQM2aNUlLS2PXrl3UqlUrb70jR47g/NbOb9WqVbRq1YqWLVtSqVIlhg0bxpw5cyJWX+Mo6gjqLBGZIyJnudPrReR1EXkNKHIEH4CqfqKqoqqdVDXd/Zuvqr9X1Y7u/IHuaL8y6VQ/0Pv372fIkCG0bduWtLQ0VqxYEc3wjckn1BFFnz59yMrKYv369bRu3Zq//e1vhdbdtWsXkyZNYvXq1WRlZZGbm8vMmTMjFltycrKnsWVnZ7N27Vp69OgBwAMPPEDTpk15/fXXgx5B7dq1i6ZNm+ZNp6amsmvXrojV1ziKOgf1F+AmYJyITAFeBv4EPK6q10Qpvrh2Kh9ogNtuu43+/fuzefNm1q1bR1paWpRrYMxJoY4o+vbtS3KyM+C3Z8+e7Ny5M+j6OTk5HD16lJycHH7++WcaN24csdhSUlI8i+3w4cMMHjyYp59+Ou/o6a9//Ss7duxgxIgRTJ48uVB5wQaXBTvSMqemuHNQR4DbgeeAKcBw4GuvgyorTuUDffDgQZYtW8aoUaMAqFSpEnXq1Ile8MYUoeARhd/LL7/M7373u0LLN2nShLvvvpvTTz+dlJQUateuTd++feM+tuPHjzN48GBGjBjBlVdeWWjda665htmzZxean5qayo4dO/Kmd+7cGdEG2TiKOgf1F+B9YAnQW1UHAutwBkn8PkrxlRkl/dBs27aNBg0acOONN9K5c2dGjx7NkSNHohWuMSEFO6IA56giOTmZESNGFFrnp59+Ys6cOWzfvp3du3dz5MgRXnvttbiOTVUZNWoUaWlp3HnnnXnLb926Ne/x3Llzadu2baEyu3XrxtatW9m+fTvHjh1j5syZDBw4MMK1NUUdQV2qqr2Ac4HrAFR1LtAPqBuF2MqM0nxocnJy+PLLL7n55ptZu3Yt1atXt6GqJuZCHVFMnz6defPm8frrrwftylq8eDEtWrSgQYMGVKxYkSuvvJLPPvssrmPLysri1Vdf5aOPPiI9PZ309HTmz5/PfffdR4cOHejUqRMffvhh3rnl3bt3M2CAkzo0OTmZyZMn069fP9LS0rj66qtp3759ROtrik51lCUirwJVcS60BUBVc4BnvA6srCjuQ7NkyZKgH5rU1FRSU1PzjriGDBliDZSJqVBHFAsXLuSxxx7j448/plq1akHXPf3001m5ciU///wzVatWZcmSJXTtGvTay7iJrWPHjkHPJfkboYIaN27M/Pnz8y0XalkTGUUNkrgWeBx4UFVtKHgQxX1o5s6dG/JDc9ppp9G0aVO2bNkCwJIlS2jXrl1U4jYmmFBHFLfccguHDh2iT58+pKenM3bsWCD/EUWPHj0YMmQIXbp0oWPHjpw4cYIxY8ZELLZPP/00bmMz3gmZ6iiexGuqo2effZbx48fTsWNHKlRw2vpHH32U8ePH8+uvv1KvXj3AGSjxwgsvsHv3bkaPHp33KywzM5PRo0dz7NgxWrZsydSpU/nNb34Ts/oUy1IdxY0dO3Zw3XXX8d1331GhQgXGjBnDbbfdxltvvcWECRPYtGkTq1atCnoUs2XLFoYOHZo3vW3bNh5++GHS09Pjtr5eiOf964V4rW9RqY68vN1GuXeqXQTp6enEY8Nr4p//uqAuXbpw6NAhzj77bPr06UOHDh14++23uemmm0Ku26ZNGzIzMwEnI0KTJk0YNGgQ27dvj1b4xoSluNttCJCqqjuKWs4YE10pKSmkpDhZwwIvcejTp0+JylmyZAlnnHEGzZo1S7gGKsN3uXOvhkiK4x6BsqjI66DUOTx4N0qxGGNKIdQlDuGYOXMmw4cP9yAqY05dOF18K0Wkm6p+4Xk0xpgSCXWJQziOHTvG3Llz8zKdxPMRRfP7Qt75p9Syq0S8SBNh4TRQvYGbROTfOJklBOfgqpOnkZUB8fyBNuVfcVkQirNgwQK6dOlCo0aNPIjOmFMXTgNVOA2CMSamQl3iUBIzZsyw7j0T14q9H5Sq/huog3P798uAOu48Y0yMhLou6J133iE1NZUVK1ZwySWX0K9fPyD/dUEAP//8M4sWLSrVkZcx0VLsEZSI3Ab8F/C2O+s1EZmiqs96GpkxJqTzzz8/6CUOAIMGDSo0r+AlDtWqVWPfvn2exWdMJITTxTcK6KGqRwBE5DFgBWANlDHGGM+Ec8t3AXIDpnPdecYYY4xnwjmCehn4XETecaevAF7yLiRjjDGm+EwSFYDPcbKZn49z5HSjqq6NQmzGGGMSWJENlKqeEJEnVfUc4MsoxWSMKYZduGoSQTjnoD4UkcES7KZGxhhjjEfCOQd1J1AdyBGRXziZSaJkeVWMMcaYEgjnHFR/Vf00SvEYY4wxQPHZzE8AT0QpFmOMMSaPnYMyxhgTl0pyDipXRI5i56CMMcZEQbENlKrWjEYgxhhjTKBiu/jEca2IPORONxWR7t6HZowxJpGFcw7qeeAc4Bp3+jDwXHEruQ3ZUhHZJCIb3KzoiEhdEVkkIlvd/78pdfTGGGPKrXAaqB6qOg74BUBVfwIqhbFeDnCXqqYBPYFxItIOuA9YoqpnAkvcaWOMMSafcBqo4yKSBCiAiDQAThS3kqruUdUv3ceHgE1AE+ByYLq72HSc5LPGGGNMPuGM4psEvAM0FJG/AkOAB0uyERFpDnTGSTzbSFX3gNOIiUjDEOuMAcYApKSk4PP5SrLJqMjwoMx4rKdfhgdlxnN9s7Oz4zq+siDRXr94rm9ZfD+HM4rvdRFZA1yEM8T8ClXdFO4GRKQGMBu4XVUPhns5lapOAaYAdO3aVTMyMsLdZPT4Il9kXNbTzxf5IuO5vj6fL37jWxj5ZLFeiNjrl2j19UBcv59DCNlAiUgNVT0MoKqbgc1FLROijIo4jdPrquq/Zfz3IpLiHj2lAD+cUg2MMcaUS0Wdg5ojIk+KSC8Rqe6fKSItRWSUiHwA9A+1spt54iVgk6r+PeCpucD17uPrgTmlD98Yb40cOZKGDRvSoUOHvHmZmZn07NmT9PR0unbtyqpVq4Kuu3DhQtq0aUOrVq2YOHFitEI2ptwI2UCp6kU4o+xuAjaIyEER2Qe8BpwGXK+q/1dE2ecBvwd+KyKZ7t8AYCLQR0S2An3caWPi0g033MDChQvzzbv33nv505/+RGZmJg8//DD33ntvofVyc3MZN24cCxYsYOPGjcyYMYONGzdGK2xjyoXiblg4H5hfmoJV9ROcc1bBXFSaMo2Jtl69epGdnZ1vnohw8OBBAA4cOEDjxo0Lrbdq1SpatWpFy5YtARg2bBhz5syhXbt2nsdsTHkRzig+Y8qEkSNHMm/ePBo2bEhWVhYAQ4cOZcuWLQDs37+fOnXqkJmZWWjd/fv3M3r0aLKyshARXn755ZDbefrpp+nXrx933303J06c4LPPPiu0zK5du2jatGnedGpqKp9//vmpVtGYhBLOdVDGlAnBuuNmzZpFZmYmmZmZDB48mCuvvDLourfddhv9+/dn8+bNrFu3jrS0tJDb+cc//sFTTz3Fjh07eOqppxg1alShZVS10Dy7IYAxJWMNlCk3evXqRd26dYM+p6q8+eabDB8+vNBzBw8eZNmyZXkNTaVKlahTp07I7UyfPj2vobvqqquCDpJITU1lx44dedM7d+4M2hVojAktrAZKRM4XkRvdxw1EpIW3YRkTWcuXL6dRo0aceeaZhZ7btm0bDRo04MYbb6Rz586MHj2aI0eOhCyrcePGfPzxxwB89NFHQcvs1q0bW7duZfv27Rw7doyZM2cycODAyFXImFIINip16NChpKenk56eTvPmzUlPTy+03o4dO+jduzdpaWm0b9+eZ555JirxhpPN/E/AH4A/urMq4ozkM6bMmDFjRtCjJ4CcnBy+/PJLbr75ZtauXUv16tXzhoUPHz6cc845hy1btpCamspLL73Eiy++yF133cVZZ53F/fffz5QpUwDYvXs3AwYMACA5OZnJkyfTr18/0tLSuPrqq2nfvn10KmtMCKXtBk9OTubJJ59k06ZNrFy5kueeey4qo1LDGSQxCCdNkT+v3m4RsXtEmTIjJyeHt99+mzVr1gR9PjU1ldTUVHr06AHAkCFDmDhxIhdddBEzZswIuk6wsho3bsz8+ScHvQ4YMCCvwTImHgQblern7wb/6KOPCj2XkpJCSkoKADVr1iQtLY1du3Z5Pio1nC6+Y+qc8fUni61ezPLGxJXFixfTtm1bUlNTgz5/2mmn0bRp07zRfkuWLLHh4CbhFNUNHig7O5u1a9fm/aDzUjgN1Jsi8k+gjoj8F7AYeNHbsIwpuWDdcQAzZ84s1L0X2B0H8OyzzzJixAg6depEZmYm999/f1RjNybWiuoG9zt8+DCDBw/m6aefplatWp7HFE6y2CdEpA9wEGgD/D9VXeR5ZMaUUKjuuGnTphWaV7A7Lj09ndWrV3sVmjFxrbhucIDjx48zePBgRowYEfJyjUgrtoFyR+wt9zdKIlJVRJqrarbXwRkTSxm+yyOfwX3CgQgXaMypK64bXFUZNWoUaWlp3HnnnVGLK5wuvrfIf4PCXHeeMcaYMqS03eCffvopr776Kh999FHekPTAHgivhDOKL1lVj/knVPWYiIRzy3djoqb5fZG/X1B2lYgXaUxMlbYb/Pzzzw+aHcVr4RxB7RWRvCsMReRy4EfvQjLGGGPCO4IaC7wuIpNxspPvAK7zNCpjjDEJL5xRfP8Cerq3bhdVPeR9WMYYYxJdOKP4KgODgeZAsj8js6o+7GlkxhhjIqYsjkoNp4tvDnAAWAP86mk0xhhjjCucBipVVft7HokxxhgTIJxRfJ+JSEfPIzHGGGMChHMEdT5wg4hsx+niE0BVtZOnkRljjElo4TRQv/M8CmOMMaaAcIaZ/xtARBoCdm29McaYqAjnjroDRWQrsB34GMgGFngclzHGmAQXziCJR4CewNeq2gK4CPjU06iMMcYkvHAaqOOqug+oICIVVHUpkO5xXMYYYxJcOIMk9rtpjpbh5OT7AcjxNixjjDGJLpwjqMuBo8AdwELgX8BlXgZljDHGhDOK7wiAiNQC3vM8ImOMMYbwRvHdJCLfA+uB1Tg5+VaHsd7LIvKDiGQFzJsgIrtEJNP9G3AqwRtjjCm/wjkHdTfQXlVLepPCacBk4JUC859S1SdKWJYxxpgEE845qH8BP5e0YFVdBvynxBEZY4wxhHcE9UechLGfE3C7DVUdX8pt3iIi1+F0E96lqj8FW0hExgBjAFJSUvD5fKXcnHcyPCgzHuvpl+FBmfFcXy9Yfcu3eK5vhgdlel3fcBqofwIfAV8BJ05xe//AufBX3f9PAiODLaiqU4ApAF27dtWMjIxT3LQHfJEvMi7r6eeLfJERq+/C9yNTjsesvqWUaPX1gi/yRXpd33AaqBxVvTMSG1PV7/2PReRFYF4kyjXGGFP+hHMOaqmIjBGRFBGp6/8rzcZEJCVgchCQFWpZY4wxiS2cI6hr3P9/DJinQMuiVhKRGTjdnvVFZCfwJyBDRNLd9bOBm0oYrzHGmARRZAMlIhWA+1R1VkkLVtXhQWa/VNJyjDHGJKYiu/hU9QQwLkqxGGOMMXnCOQe1SETuFpGmp3oOyhhjjAlXOOeg/MPAA4+kij0HZYwxxpyKcJLFtohGIMYYY0ygYhsoEakI3Az0cmf5gH+q6nEP4zLGGJPgwuni+wdQEXjenf69O2+0V0EZY4wx4TRQ3VT1rIDpj0RknVcBGWOMMRDeKL5cETnDPyEiLYFc70IyxhhjwjuCugcn3dE2QIBmwI2eRmWMMSbhhWygROQqVX0L2AacCbTBaaA2q+qvodYzxhhjIqGoLj5/7r3Zqvqrqq5X1XXWOBljjImGorr49onIUqCliMwt+KSqDvQuLGOMMYmuqAbqEqAL8CrOjQWNMcaYqAnZQKnqMRH5Aliuqh9HMSZjjDGm2GzmuTij9owxxpioCmeYeaZ7Duot4Ih/pqq+7VlUxhhjEl44DVRdYB/w24B5ClgDZYwxxjPhZDO3i3KNMcZEXbGpjkSktYgsEZEsd7qTiDzofWjGGGMSWTi5+F7EuWj3OM6/sCEAABnhSURBVICqrgeGeRmUMcYYE04DVU1VVxWYl+NFMMYYY4xfOA3Uj242cwUQkSHAHk+jMsYYk/DCGcU3DpgCtBWRXcB2YISnURljjEl44Yzi2wZcLCLVgQqqesj7sIwxxiS6cEbx1RORScBywCciz4hIPe9DM8YYk8jCOQc1E9gLDAaGuI9neRmUMcYYE1YmCVV9JGD6LyJyhVcBGWOMMRDeEdRSERkmIhXcv6uB970OzBhjTGILp4G6CXgD+NX9mwncKSKHROSgl8EZY4xJXMU2UKpaU1UrqGpF96+CO6+mqtYKtZ6IvCwiP/hTJLnz6orIIhHZ6v7/TaQqYowxpnwJ5wiqtKYB/QvMuw9YoqpnAkvcaWOMMaYQzxooVV0G/KfA7MuB6e7j6YANtjDGGBNUOKP4IqmRqu4BUNU9ItIw1IIiMgYYA5CSkoLP54tOhCWQ4UGZ8VhPvwwPyozn+nrB6lu+xXN9Mzwo0+v6hmygRKRuUSuqasGjo4hS1Sk4KZbo2rWrZmRkeLm50vFFvsi4rKefL/JFRqy+C8vGwFKrbyklWn294It8kV7Xt6gjqDU4CWIlyHMKtCzF9r4XkRT36CkF+KEUZRhjjEkAIRsoVW3hwfbmAtcDE93/czzYhjHGmHIgnFx8IiLXishD7vTpItI9jPVmACuANiKyU0RG4TRMfURkK9DHnTbGGBPCli1bSE9Pz/urVasWTz/9dL5lDhw4wGWXXcZZZ51F+/btmTp1aoyijaxwBkk8D5wAfgs8AhwCZgPdilpJVYeHeOqikgRojDGJrE2bNmRmZgKQm5tLkyZNGDRoUL5lnnvuOdq1a8d7773H3r17adOmDSNGjKBSpUqxCDliwmmgeqhqFxFZC6CqP4lI2a61McaUQUuWLOGMM86gWbNm+eaLCIcOHUJVOXz4MHXr1iU5OdqDtCMvnBocF5EkTt5RtwHOEZUxxpgomjlzJsOHF+6cuuWWWxg4cCCNGzfm0KFDzJo1iwoVvMzDEB3h1GAS8A7QUET+CnwCPOppVMYYY/I5duwYc+fO5aqrrir03AcffEB6ejq7d+8mMzOTW265hYMHy36q1HBy8b0O3Av8DdgDXKGqb3kdmDHGmJMWLFhAly5daNSoUaHnpk6dypVXXomI0KpVK1q0aMHmzZtjEGVkhWyg3MSudd0Ldn8AZuBkNf++uIt4jTHGRNaMGTOCdu8BnH766SxZsgSA77//ni1bttCyZWkuVY0v4V6oezrwk/u4DvAt4MV1UsYYYwr4+eefWbRoEf/85z/z5r3wwgsAjB07loceeogbbriBjh07oqo89thj1K9fP1bhRkyxF+qKyAvAXFWd707/Drg4OuEZY4ypVq0a+/btyzdv7NixeY8bN27Mhx9+GO2wPBfOIIlu/sYJQFUXABd6F5IxxhgT3jDzH0XkQeA1nC6/a4F9Ra9ijDHGnJpwjqCGAw1whpq/CzR05xljjDGeKfYIyr2txm0iUgs4oaqHvQ/LGGMSV/P7In97kewqES/Sc+Eki+3opjn6CtggImtEpIP3oRljjElk4XTx/RO4U1WbqWoz4C7cGwkaY4wxXgmngaquqkv9E6rqA6p7FpExxhhDeKP4trn3gnrVnb4W2O5dSMYYY0x4R1AjcUbxvY0zkq8BcKOXQRljjDHhjOL7CRgfhViMMcaYPCEbKBGZW9SKqjow8uEYY4wxjqKOoM4BduBkMf8cJ1GsMcYYExVFNVCnAX1wskZcA7wPzFDVDdEIzBhjTGILOUhCVXNVdaGqXg/0BL4BfCJya9SiM8YYk7CKHCQhIpWBS3COoprj3P79be/DMsYYk+iKGiQxHegALAD+rKpZUYvKGGNMwivqCOr3wBGgNTBeJG+MhACqqrU8js0YY0wCK+qOuuFcxGuMMcZ4whohY4wxcckaKGOMMXHJGihjjDFxKZxs5hEnItnAISAXyFHVrrGIwxhjTPyKSQPl6q2qP8Zw+8YYY+KYdfEZY4yJS7E6glLgQxFR4J+qWugW8iIyBhgDkJKSgs/ni26EYcjwoMx4rKdfhgdlxnN9vWD1Ld+svpEVqwbqPFXdLSINgUUisllVlwUu4DZaUwC6du2qGRkZMQizGL7IFxmX9fTzRb7IiNV34fuRKcdjVt9SsvrGJa+/r2LSxaequ93/P+Dcpbd7LOIwxhgTv6LeQIlIdRGp6X8M9AUsz58xxph8YtHF1wh4x83tlwy8oaoLYxCHMcaYOBb1BkpVtwFnRXu7xhhjyhYbZm6MMSYuWQNljDEmLlkDZYwxJi5ZA2WMMSYuWQNljDEmLlkDZYwxJi5ZA2WMMSYuWQNljDEmLlkDZYwxJi5ZA2WMMSYuWQNljDEmLlkDZYwxJi5ZA2WMMSYuWQNljDEmLlkDZYwxJi5ZA2WMMSYuWQNljDEmLlkDZYwxJi5ZA2WMMSYuJWwDtXDhQtq0aUOrVq2YOHFioedVlfHjx9OqVSs6derEl19+GYMoIyfR6muMKfsSsoHKzc1l3LhxLFiwgI0bNzJjxgw2btyYb5kFCxawdetWtm7dypQpU7j55ptjFO2pS7T6GmPKh4RsoFatWkWrVq1o2bIllSpVYtiwYcyZMyffMnPmzOG6665DROjZsyf79+9nz549MYr41CRafY0x5UNCNlC7du2iadOmedOpqans2rWrxMuUFYlWX2NM+ZCQDZSqFponIiVepqxItPoaY8qHhGygUlNT2bFjR970zp07ady4cYmXKSsSrb7GmPIhIRuobt26sXXrVrZv386xY8eYOXMmAwcOzLfMwIEDeeWVV1BVVq5cSe3atUlJSYlRxKcm0eprjCkfkmMdQCwkJyczefJk+vXrR25uLiNHjqR9+/a88MILAIwdO5YBAwYwf/58WrVqRbVq1Zg6dWqMoy69RKuvMaZ8SMgGCmDAgAEMGDAg37yxY8fmPRYRnnvuuWiH5ZlEq68xpuyLSRefiPQXkS0i8o2I3BeLGIwxxsS3qDdQIpIEPAf8DmgHDBeRdtGOwxhjTHyLxRFUd+AbVd2mqseAmcDlMYjDGGNMHJNg1794ukGRIUB/VR3tTv8e6KGqtxRYbgwwxp1sA2yJaqDhqQ/8GOsgosjqW75Zfcu3eK1vM1VtEOyJWAySCHb1Z6FWUlWnAFO8D6f0RGS1qnaNdRzRYvUt36y+5VtZrG8suvh2Ak0DplOB3TGIwxhjTByLRQP1BXCmiLQQkUrAMGBuDOIwxhgTx6LexaeqOSJyC/ABkAS8rKoboh1HhMR1F6QHrL7lm9W3fCtz9Y36IAljjDEmHAmZi88YY0z8swbKGGNMXEqoBkpEckUkU0SyROQ9EakToXJvEJHJkSgrmgJeD/9fkWmnROT+aMV2KkTkcATK6Coik4p4vrmIXBPu8pEWsO/WiciXInJutLYdJJYMEZnnPs77LIjIWBG5LkYxPSAiG0Rkvfs6LRCRvxVYJl1ENrmPs0VkeYHnM0UkK0LxBN1f7vsoUtsI3A8Do5VGLnA/u/u/ccBz2SJSv7RlJ1qy2KOqmg4gItOBccBfYxtSTOW9HmG6H3i04Exx7mwoqnoiYpHFmKquBlYXsUhz4BrgjTCXj7TA93I/4G/AheGsGK39paoveFl+KCJyDnAp0EVVf3W/INsDU4E/Biw6DHf/uWqKSFNV3SEiaREOq9T7qzRUdS5RGh1dYD/fAGQRoUuHEuoIqoAVQBMAEekuIp+JyFr3fxt3/g0i8raILBSRrSLyuH9lEblRRL4WkY+B8wLmNxORJe4vtyUicro7f5qI/ENElorINhG5UEReFpFNIjItqjUvgojUdhP5+l+DGSLyXyIyEajq/gp83f3lt0lEnge+BJq69Vvt/nL9c0wrUoD7a3mlu1/eEZHfuPO7ufNWiMj/+H/NFvg1emHAUeZaEakJTAQucOfdUWD5GiIyVUS+csse7HH1agE/BdT1HhH5wt32n915BffXBe70i+7++lBEqhbzWvlEpKv7uL6IZBcVlIhMEJG7valykVKAH1X1VwBV/VFVPwb2i0iPgOWuxkm15vcmMNR9PByY4VF8+faXn4gkue9B/767yZ2f4b72/ycim93Pn7jP9XfnfQJcGVBW4JHsNBGZ5H63bRMnmw8iUkFEnnf3/zwRme9/LqCchiKyxn18lohowHfav0Skmn8/u+t2BV53PxdV3WJuFeeo8SsRaVuiV0pVE+YPOOz+TwLewkm55H/DJLuPLwZmu49vALYBtYEqwL9xLjJOAb4FGgCVgE+Bye467wHXu49HAu+6j6fhfBgEJ/fgQaAjzo+ENUB6DF6PXCAz4G+oO78PTgM+DFhY8PVzHzcHTgA9A+bVDXh9fUCnWO7nAvPWAxe6jx8GnnYfZwHnuo8nAlnu4wxgXsA+Pc99XAOn5yHv+SDLP+Yv353+jYf7bjNwADjbnd8XZzixuO+teUCvgvvLnc7xv+9wvpyvLea18gFd3cf1gewgdb8h4LMwAbg7Bvu/hvvafA08H1CXe4Cn3Mc9gS8C1skGWgOfudNrcZJZZ3m8v5oHvOfGAA+6jyvjHJG3cF/fAzhJDSrgfDbPx/lO2gGc6e7vN0Psh2k433cV3Dp9484fAsx355+G02gOCRL7BpzvyFtwrmMdATQDVhTcz4HvkYDX9Vb38X8D/1uS1y3RjqCqikgmsA+oCyxy59cG3nJ/PT+F0x3gt0RVD6jqL8BGnB3TA/Cp6l51Et7OClj+HE52G7yK80bye0+dPfUV8L2qfqVON8sGnDdqtB1V1fSAv1kAqrrIjfE5YHQR6/9bVVcGTF8tIl/ifLjb43wYYk5EagN11PkVDTAd6CXOOciaqvqZO/+NoAU4P0D+LiLj3XJyitnkxTivHQCqWujXcgT4911boD/wivuruq/7txbnSKktzhcYFN5f21U10328Bmge6rXyIH7PqOph4GycL/y9wCwRuQHnB+IQEamA8+Or4BHSf4CfRGQYsAn4OYJhhdpfgfoC17nfUZ8D9Ti571ap6k73+yIT5/uiLc4+3Op+r7xWxPbfVdUTqroRaOTOOx94y53/HbA0xLqf4fQS9cLp4u8FXAAsD7F8QW+7/9dQwu+5RGug/P3AzXCOfMa58x8BlqpqB+AynF8mfr8GPM7l5Hm7cC8gC1zOX9aJAuWeII7OB7of4DTgKE5DHsqRgHVaAHcDF6lqJ+B98r+O8ShYXshCVHUiTkNdFVgZRjeFEP7745Sp6gqcI5oG7rb/FvCjo5WqvuQueqTAqqHe26HkcPI7I673rarmqqpPVf+E88t/sKruwPlFfyEwGOeIo6BZOD8uvOreK7i/AgnO0YZ/37VQ1Q/d5071eyhwfSnwvzjLcRqkZsAc4Cycxm1ZCbcdznssn0RroABQ1QPAeOBuEamIcwS1y336hjCK+BzIEJF67vpXBTz3Gc6vM3AOhT+JSNDRdQfOL8jhwMtuHQGOBzwuqBbOF+ABEWmEc7+vuODu759E5AJ31u+Bj90jm0Mi0tOdPyzY+iJyhnu0+xhOt0tb4BBQM8QmP8T5UvSv/5sIVCMkt8FMwukZ+AAYKSI13OeaiEjDcMsK9Vq5j7NxjkzA6R6KSyLSRkTODJiVjtM9D07D8xTwL1XdGWT1d4DHcV5Hr+IL3F+BPgBu9n/GRKS1iFQvoqjNQAsROcOdHl7CUD4BBrvnohrhdCUGswy4FtjqHsH9BxiA07NQUFGfixKLm1/t0aaqa0VkHc6X0uPAdBG5E/gojHX3iMgEnL7gPThdKUnu0+NxvtTvweleuNGD8CPF3+XptxB4GedoobuqHhKRZcCDwJ9wzm2sd7vxHggsSFXXichanO7KbQR/80ZLNREJ/PL5O3A98IKIVMOJz79fRgEvisgRnP7zA0HKu11EeuP8AtwILMA56s1x30PTcLrU/P4CPOd2GecCf+ZkN0ekBO47wTnvmQt8KM4ItBVuD9JhnC+X3BKUHeq1egJ4U5xb5BT7OYmhGsCzbhduDvANJ2/d8xbwDHBrsBVV9RDOOUQK98CdkqD7q8A2/henC+xLt/tvL3BFqAJV9Rdxbkv0voj8iNPgdChBTLOBi3DOw36N88O70PtfVbPdOP1HTJ8AqSG6rqfhvHeO4pzuOCWW6sgkNBGp4Z6zQJzrRlJU9bYYh2VMVPjf/yJSD1iFMxjou1jH5ZewR1DGuC4RkT/ifBb+TXhdvMaUF/PcI81KwCPx1DiBHUEZY4yJUwk5SMIYY0z8swaKoDnpmnu0nQyJYc60QO4IRH99vxORXQHTlWIdX1HEo5yKkSAiD4vIxREqK1tEZgdMD5Fiso5IhHICuu/VA+7rvF5EFpdkNKApnpuV4cmA6bvdwVdebzcvI0iQ+asDpruKiK+YsvLlpIw0a6AcBS9YzQ5nJREp6Tm8DCAuGihV3eevL/ACzhX2/vofK0Xdosm/vzrgDHkdV9wKxRGRpOKXKp6q/j9VXRyJslxdRaR98YvlbX+1qo6P0LaXu69zJ5wMAqf8Opt8fgWulFNIphqMOEr73d5QREpyiUhznJyUnrAGKgQpOh/Zo+Lk4LtNRM4WkY9FZI2IfCAiKe5y40Vko7v+TPeobCxwh/ur9IKQG48RcXJ2/V1ElgKPicgZ4uQhXCMiy8W9QFVEGojIbHFyhn0hIue584PlrPNaYE7FUPGe4e7LL9wjHP+ovQxxciO+AXwloXOhpYjIsoCjtgvcZae501+JyB0Br6E/19lF7uvwlTh5Fyu787NF5M8SXn6yJ3CS9OYjofNHZoiTV62Cu506Aet8IyKNQu2/UEREcK5t+amYbS8XkfSA9T4VkU4iUt2t/xfuOpe7z7cXkVVy8ijtzGDbL8dycC7duKPgE0V8xvLlN3Tff80lcrkx/wfnspKC8QT9bFA4J2Vk92lJ8iKV1z/y56R7x51XVD6y593HFXEuzG3gTg/FuYU9ONl8K7uP62iBnFXx9OePC+cahnlAkjt/CXCm+7gH8JH7+A3gfPfx6cAm93GhnHUexRsqp2KoeOcBw93HYwPWz8C5uLiFOx0qF9pdwAMB26yJc8HqooCY/Pt4Gs5FrP48aa3d+a8At7uPswkjP5m7XCOci6ZbueVOc58LlT8yg5P52J4Bbgx4PRYXtf8KbDsD55qYTLcem4FaxWz7ek5+TloDq93Hj3Iyz18dnGtuqgPPAiPc+ZWAqrH+LET5c3fYfS2zcZIF3A1MKOYzNoGA7xCca5iaU4LcmBTIlxewvA8n2etHQG/3sa+Yz0be+819LqL7NJ67caIp320nJHg+srcClvfn3muDc2HcIudHJkk4F+6C08C9LiLvAu96GHukvaXOBYQ1cLoj35KTFxNWdv9fDLQLmF/LPVry56x7HXhbg1+pHwn+ix6b4+T3WlRMvOdw8oLHN3COSvxWqep293FfoJOczOhcGycX2heczKjxrqpmisg2oKWIPIuT1ulD8muDkyfta3faf3uXp93pwPxkVxJaLs6v2j/iXCDsVxvn4vIzcdLdBMvwMQv4fzi3mRjGyfdt0P2nzkWqgZar6qUAIvIHnAvaxxax7beAh8S5SH0kTmMNzus6MOCXfxWcL90VwAMikorzftlaxOtQLqnqQRF5BecC/6MBT4X6jBUlWG7MMTiXUKTg5MZcH0ZYf8E5ivpDwLxQn41jBdaN6D61Bqp0/DnNBNigqsGumL4EJ6niQJwPbdjnEWLMX7cKwH4Nfr+oCsA5qnq0wPyJIvI+ThqUlSJysapu9iDGo6qa7v6QmIfzxT+tiHiLEpifzp8LrVCaGxHphbNPXxWR/1HVV0TkLKCfu/2rcb6UA8sqSknyk72K00BtCJjnzx85SJzuY1+Q9VYArUSkAU4D/Rd3fqj9V5S5OJkHQm5bVX8WkUU42fqvxvkFDs5rMVhVtxQoc5OIfI7zun4gIqNVNZ4zVHjlaZxuuakB84LuIxEJzIcI+XMiBsuN2U1VfxJncE1Y+RNV9SMReQQn43tekQT5bIhIRoF134jkPrVzUEFo0fnIAm0BGohzgzREpKLbB1sBaKqqS4F7cbo1ahDhPFVeUtWDwHYRuQryTrye5T5dMNec/0ZswXLWeRljXk5FnF+foeJdiZMcFELk23MFzYUmIs2AH1T1ReAloIs4J7YrqOps4CGgS4GyNuNkB2/lTod6D4VTz+M4+eNuD5hdbP5IdfpZ3sFJ9bRJVf2534Luv2KcD/wrjG3/LzAJ51YW/3HnfYBzTyD/PYw6u/9bAttUdRJOA9gpjDjKHfd1ehMn7ZZfqH2UjfteE5EuON1swZxqbsy/4nx3+YXKE5jvOy3S+9QaqNCuB/5HRNbjJJt8uOAC6txqYwjOgIJ1OP315+J09b0mIl/h5Gh7SlX345yjGSRxOkgiiBHAKLduG3B+GYPTKHR1T4JuxOn2ASdnXZa7/FHyd0l5QlXXAv6ciqHivR24U0RW4XR1BMu3B86X60acXGhZwD85ee+nTHFyDQ7GObfTBPC5XY3TyH+nVtS5PcuNOF2OX+GcHziVO8y+RP4jrceBv4nIp5zMAxnMLJxcfIG3hAm1/wryn/xeh9PA3lXctlV1Dc69zgKPBh7B6QZc776uj7jzhwJZ7mvYFuc8XaJ6EifDuV+ofTQbqOu+ZjfjnM8rRFXX4Xz3bMDJr1mi3JiqOh8nF6BfqM/GetyclOIMFIroPrVMEqbcEyfp6VFVVXHu9TNcVS8vbj1TciLSGKfLr616fEt5U/7ZOSiTCM4GJrtdTPvJf67IRIiIXIfTNXSnNU4mEuwIyhhjTFyyc1DGGGPikjVQxhhj4pI1UMYYY+KSNVDGGGPikjVQxhhj4tL/BxT6y+7kyyIxAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "labels = ['Random\\nForest', 'Extra\\nTrees', 'Logistic\\nRegression', \n",
    "      'Bernoulli\\n Naive Bayes', 'SVM', 'Blending with\\nNeural Nets']\n",
    "\n",
    "x = np.arange(len(labels))  # the label locations\n",
    "width = 0.35  # the width of the bars\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "rects1 = ax.bar(x - width / 2, trn_errors, width, label='Training Error')\n",
    "rects2 = ax.bar(x + width / 2, tst_errors, width, label='Test Error')\n",
    "\n",
    "# Add some text for labels, title and custom x-axis tick labels, etc.\n",
    "ax.set_ylabel('Model performance (error %)')\n",
    "# ax.set_title('Scores by group and gender')\n",
    "ax.set_xticks(x)\n",
    "ax.set_xticklabels(labels)\n",
    "ax.legend()\n",
    "\n",
    "ax.set_axisbelow(True)\n",
    "ax.grid(linestyle='-', linewidth='0.5', color='gray')\n",
    "\n",
    "autolabel(ax, rects1)\n",
    "autolabel(ax, rects2)\n",
    "\n",
    "fig.tight_layout()\n",
    "plt.savefig('./figures/CH03_F19_Kunapuli.png', dpi=300, bbox_inches='tight', pad_inches=0)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}