Bleu_Score.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculating the Bilingual Evaluation Understudy (BLEU) score: Ungraded Lab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this ungraded lab, we will implement a popular metric for evaluating the quality of machine-translated text: the BLEU score proposed by Kishore Papineni, et al. In their 2002 paper [\"BLEU: a Method for Automatic Evaluation of Machine Translation\"](https://www.aclweb.org/anthology/P02-1040.pdf), the BLEU score works by comparing \"candidate\" text to one or more \"reference\" translations. The result is better the closer the score is to 1. Let's see how to get this value in the following sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 1:  BLEU Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1  Importing the Libraries\n",
    "\n",
    "We will first start by importing the Python libraries we will use in the first part of this lab. For learning, we will implement our own version of the BLEU Score using Numpy. To verify that our implementation is correct, we will compare our results with those generated by the [SacreBLEU library](https://github.com/mjpost/sacrebleu). This package provides hassle-free computation of shareable, comparable, and reproducible BLEU scores. It also knows all the standard test sets and handles downloading, processing, and tokenization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to /home/jovyan/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sacrebleu in /opt/conda/lib/python3.7/site-packages (1.4.12)\n",
      "Requirement already satisfied: portalocker in /opt/conda/lib/python3.7/site-packages (from sacrebleu) (1.7.1)\n",
      "Requirement already satisfied: mecab-python3==0.996.5 in /opt/conda/lib/python3.7/site-packages (from sacrebleu) (0.996.5)\n",
      "\u001b[33mWARNING: You are using pip version 20.1.1; however, version 20.2.3 is available.\n",
      "You should consider upgrading via the '/opt/conda/bin/python3 -m pip install --upgrade pip' command.\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "import numpy as np                  # import numpy to make numerical computations.\n",
    "import nltk                         # import NLTK to handle simple NL tasks like tokenization.\n",
    "nltk.download(\"punkt\")\n",
    "from nltk.util import ngrams\n",
    "from collections import Counter     # import the Counter module.\n",
    "!pip3 install 'sacrebleu'           # install the sacrebleu package.\n",
    "import sacrebleu                    # import sacrebleu in order compute the BLEU score.\n",
    "import matplotlib.pyplot as plt     # import pyplot in order to make some illustrations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2  Defining the BLEU Score\n",
    "\n",
    "You have seen the formula for calculating the BLEU score in this week's lectures. More formally, we can express the BLEU score as:\n",
    "\n",
    "$$BLEU = BP\\Bigl(\\prod_{i=1}^{4}precision_i\\Bigr)^{(1/4)}$$\n",
    "\n",
    "with the Brevity Penalty and precision defined as:\n",
    "\n",
    "$$BP = min\\Bigl(1, e^{(1-({ref}/{cand}))}\\Bigr)$$\n",
    "\n",
    "$$precision_i = \\frac {\\sum_{snt \\in{cand}}\\sum_{i\\in{snt}}min\\Bigl(m^{i}_{cand}, m^{i}_{ref}\\Bigr)}{w^{i}_{t}}$$\n",
    "\n",
    "where:\n",
    "\n",
    "* $m^{i}_{cand}$, is the count of i-gram in candidate matching the reference translation.\n",
    "* $m^{i}_{ref}$, is the count of i-gram in the reference translation.\n",
    "* $w^{i}_{t}$, is the total number of i-grams in candidate translation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3 Explaining the BLEU score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Brevity Penalty (example):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3dd3gU5fr/8fedSmihBenSQVpAFhAQxI4NRECwg5WDgOixH79H/R2Px14ARRCVIoL9HLCBBRQpQuggLQJCbBSV3rl/f8yga9gkm5DJ7Gbv13XtlS3PzHx2srP3TntGVBVjjDGxK87vAMYYY/xlhcAYY2KcFQJjjIlxVgiMMSbGWSEwxpgYl+B3gPyqVKmS1q5d2+8YxhgTVRYuXLhNVdNCvRZ1haB27dpkZGT4HcMYY6KKiHyf02u2acgYY2KcFQJjjIlxVgiMMSbGWSEwxpgYZ4XAGGNinGeFQEReFZEtIrIih9dFRIaJSKaILBORU73KYowxJmderhGMBbrm8voFQAP3djMw0sMsxhhjcuDZeQSq+pWI1M6lSXdgvDr9YM8TkXIiUlVVf/Iiz5qfd/Hhsh+9GHWxlFoyiSvb1iIlKd7vKMYYj/l5Qll1YHPQ4yz3ueMKgYjcjLPWQK1atQo0scwtuxk+I7NAw8YiVZg473uevjydVrXK+x3HGOMhPwuBhHgu5FVyVHU0MBogEAgU6Eo6F7WoykUtLirIoDFpTuY27npnGT1HzuG5vq3oll7N70jGGI/4edRQFlAz6HENwLbdRIgO9Svx8dBOnFqrPPe9u4wN2/b4HckY4xE/C8EU4Fr36KHTgB1e7R8wBVO2RCLDr2xFYkIct05cxP5DR/yOZIzxgJeHj04C5gKNRCRLRG4QkQEiMsBt8hGwHsgEXgYGepXFFFzV1BSe6pXOtz/t5D8frfI7jjHGA14eNXRFHq8rcKtX0zeF55wmJ3F9xzq8OnsD7etVpGuzqn5HMsYUIjuz2ITl3gsak14jlbveWcbmX/f6HccYU4isEJiwJCXEMeJK5+TvQZMWc/DwUZ8TGWMKixUCE7aaFUryRM8WLN38O49/strvOMaYQmKFwOTLBc2r0q9DbV75egPTVv7sdxxjTCGwQmDy7b4Lnf0Fd7691PYXGFMMWCEw+ZacEM+IK09FgIETF3HgsJ1fYEw0s0JgCqRmhZI82Tud5T/s4JEP7PwCY6KZFQJTYOc3rcJNneowYd73TFlqvYMYE62sEJgTcnfXxgROLs+97y4jc8suv+MYYwrACoE5IYnxzvkFKYnxDHh9EXsOHPY7kjEmn6wQmBNWJbUEw65oxfqtu7nvveU4vYcYY6KFFQJTKDrWr8Qd5zZkytIfmTDve7/jGGPywQqBKTQDu9TnrMaV+dcH37Jo029+xzHGhMkKgSk0cXHCs5e3pEpqCQa+vohtuw/4HckYEwYrBKZQpZZMZORVrflt70EGv7GYw0esczpjIp0VAlPomlVP5ZFLmzF3/XaenL7G7zjGmDxYITCe6B2oyVXtajHqy/V8vNyuQGpMJLNCYDzzz0ua0LJmOe58eynrfrGTzYyJVFYIjGeSE+IZefWppCTFc8uEhezcf8jvSMaYEKwQGE9VTU1hxJWn8v2ve7njzaUcPWonmxkTaawQGM+dVrciD1x0Cp+t+oXhX2T6HccYk40VAlMk+nWozWWtqvPsZ2v5fNUvfscxxgTxtBCISFcRWSMimSJyb4jXy4vI+yKyTETmi0gzL/MY/4gIj17WnGbVyzJ08hIyt+z2O5IxxuVZIRCReOAF4AKgCXCFiDTJ1ux+YImqtgCuBZ73Ko/xX4nEeEZdEyApIY6bJ2TYzmNjIoSXawRtgUxVXa+qB4HJQPdsbZoAnwOo6mqgtoic5GEm47Pq5VJ48apT2bR9L7dPXmI7j42JAF4WgurA5qDHWe5zwZYClwGISFvgZKCGh5lMBGhXtyL/vKQJn6/ewjOfrvU7jjExz8tCICGey/7z7zGgvIgsAQYDi4HjrmwiIjeLSIaIZGzdurXwk5oid81pJ9O3TU1GzMjkg2V2mUtj/JTg4bizgJpBj2sAf1niVXUn0B9ARATY4N7I1m40MBogEAjYtoRiQER4uHtT1m3ZzZ1vL6V2xVI0q57qdyxjYpKXawQLgAYiUkdEkoC+wJTgBiJSzn0N4EbgK7c4mBiQnBDPS1e3pnzJJG4en2HdVhvjE88KgaoeBgYB04BVwFuqulJEBojIALfZKcBKEVmNc3TRbV7lMZEprUwyo68JsH3PQQZMWMiBw0f8jmRMzJFou75sIBDQjIwMv2OYQjZ16Y8MnrSY3q1r8ESvFjhbCo0xhUVEFqpqINRrXu4jMCZsl6RXY90vuxj2RSaNqpThxk51/Y5kTMywLiZMxBh6TkO6Nq3Cox+tYsbqLX7HMSZmWCEwESMuTnimTzqNq5Rl8KTFrLVrGBhTJKwQmIhSMimBMdcFSEmK54ZxC9huRxIZ4zkrBCbiVCuXwsvXBtiy8wC32JFExnjOCoGJSC1rluOp3ulkfP8b9727nGg7us2YaGJHDZmIdUl6NdZv3cOzn62lblopBp3VwO9IxhRLVghMRBtydn02bt/DU9PXUrtSKS5uUc3vSMYUO7ZpyEQ0EeGxns0JnFyeO95ayqJNv/kdyZhixwqBiXjJCfGMuqY1VcqW4KZxGWz+da/fkYwpVqwQmKhQsXQyr/Vvw+GjSr/X5rNjr13dzJjCYoXARI16aaUZdU1rNv26lwGvL+Tg4aN+RzKmWLBCYKLKaXUr8kSvFsxdv51731tmh5UaUwjsqCETdXq0qsGm7ft49rO11CxfktvPbeh3JGOimhUCE5WGnF2frN/28vzn66hePoXLAzXzHsgYE5IVAhOVRIRHL2vOzzv3c/97y6maWoJODdL8jmVMVLJ9BCZqJcbH8eJVp1K/cmn+9voiVv64w+9IxkQlKwQmqpUpkchr/dtQpkQC/V9bQNZvdo6BMfllhcBEvaqpKYzt35Z9h47Q77UFdo6BMflkhcAUC42qlGH0NQE2bd/LTeMz2H/Iuq42JlxWCEyx0b5eRZ6+PJ35G39l6OQlHDlq5xgYEw4rBKZYuSS9Gv93cRM+WfkzD01ZaSecGRMGO3zUFDs3nF6HLTv3M+qr9ZxUNtmuY2BMHjxdIxCRriKyRkQyReTeEK+nishUEVkqIitFpL+XeUzsuKdrY3q0qs5T09cyef4mv+MYE9E8WyMQkXjgBeBcIAtYICJTVPXboGa3At+q6iUikgasEZGJqnrQq1wmNsTFCY/3bMH2PQe5//3llC+VxPlNq/gdy5iI5OUaQVsgU1XXu1/sk4Hu2dooUEZEBCgN/Aoc9jCTiSFJCXGMvOpUmtcox+BJi/lm/Xa/IxkTkfIsBO4v+4KoDmwOepzlPhdsBHAK8COwHLhNVY/rW1hEbhaRDBHJ2Lp1awHjmFhUKjmB1/q1oWb5FG4cl8G3P+70O5IxESecNYJMEXlSRJrkc9wS4rnsh3CcDywBqgEtgREiUva4gVRHq2pAVQNpadafjMmfCqWSGH9DO0qXSODaV+fz/fY9fkcyJqKEUwhaAGuBMSIyz/11ftyXdQhZQHCXkDVwfvkH6w+8p45MYAPQOIxxG5Mv1culMOGGthw5epSrX/mGX3bu9zuSMREjz0KgqrtU9WVV7QDcDTwI/CQi40Skfi6DLgAaiEgdEUkC+gJTsrXZBJwNICInAY2A9QV4H8bkqX7lMozt35btuw9y7Svz+X2vHZNgDIS5j0BEuonI+8DzwNNAXWAq8FFOw6nqYWAQMA1YBbylqitFZICIDHCb/QvoICLLgc+Be1R12wm9I2NykV6zHC9fG2DDtj30H7uAvQft2ARjJK8zL0VkPTADeEVV52R7bZiqDvEw33ECgYBmZGQU5SRNMfTJip8YOHERHetXYsx1AZITCnpMhDHRQUQWqmog1Gvh7CO4VlVvCC4CItIRoKiLgDGFpWuzqjzeswWz1m1jyKTFHD5y3MFqxsSMcArBsBDPDS/sIMYUtd6Bmvzz4iZMW/kLd7+zjKPWSZ2JUTmeWSwi7YEOQJqI3BH0UlnA1qNNsXD96XXYtf8wz362ltIlEni4W1Oc8xuNiR25dTGRhHO2bwJQJuj5nUAvL0MZU5SGnF2f3QcO8fKsDZRMSuCero2sGJiYkmMhUNUvgS9FZKyqfl+EmYwpUiLC/Reewp6DR3jpy+8olRTP4LOtx1ITO3LbNDQV90zgUL+OVLWbd7GMKVoiwiPdm7H/4BGe/nQtKUnx3Niprt+xjCkSuW0aeqrIUhgTAeLihCd6tWDfoSM88uEqUpLiuardyX7HMsZzeW0aMiamJMTH8XzfVhx4fSH/eH8FyQnx9Gpdw+9YxngqnDOLG4jIOyLyrYisP3YrinDG+CEpIY4XrzqVTg0qcfc7S/nfkh/8jmSMp8I5j+A1YCTOdQLOBMYDE7wMZYzfSiTGM/qaAG1qV+COt5by8fKf/I5kjGfCKQQpqvo5TncU36vqQ8BZ3sYyxn8pSfG82q8NLWs6F7b59Ntf/I5kjCfCKQT7RSQOWCcig0SkB1DZ41zGRIRSyQm81r8NTaunMnDiQj5fZcXAFD/hFIKhQElgCNAauAa4zstQxkSSsiUSGX99W06pWpa/vb6IGau3+B3JmEIVzvUIFqjqblXNUtX+qnqZqs4rinDGRIrUlEQmXN+OhlVKc8vrC/lyrV0y1RQf4Rw11FBEXhaR6SLyxbFbUYQzJpKklkzk9RvaUT+tNDeNz+ArKwammAhn09DbwCLgAeCuoJsxMadcySQm3tiOemmluXF8hq0ZmGIhnEJwWFVHqup8VV147OZ5MmMiVPlSSbzhFoObxmcwc43tMzDRLZxCMFVEBopIVRGpcOzmeTJjItixYlA/rTQ3j1/IDCsGJoqFUwiuw9kUNAdY6N7sWpEm5pUvlcQbN7WjwUmluWX8Qr5YbYeWmugUzlFDdULcrFtGY/hzn0GjKmW4ZcJCpq/82e9IxuRbOEcNlRSRB0RktPu4gYhc7H00Y6JDuZJJvH5jO5pWS2XgxEV8ZN1RmCgTbl9DB3EuWwmQBTziWSJjolBqSiITbmhLutsdxZSlP/odyZiwhVMI6qnqE8AhAFXdB4R1HT8R6Soia0QkU0TuDfH6XSKyxL2tEJEjtiPaRKsy7hnIgZPLM3TyYt5dmOV3JGPCEk4hOCgiKfx5tbJ6wIG8BhKReOAF4AKgCXCFiDQJbqOqT6pqS1VtCdwHfKmqv+bzPRgTMUolJzC2f1s61KvE399eysRv7CqvJvKFUwgeBD4BaorIROBz4O4whmsLZKrqelU9CEwGuufS/gpgUhjjNSaipSTFM+a6AGc3rsw/3l/BmFl2+Q4T2cIpBIuBnkA/nC/qgKrODGO46sDmoMdZ7nPHEZGSQFfg3Rxev1lEMkQkY+tWO5PTRL4SifGMvLo1FzWvyiMfrmL45+tQVb9jGRNSbhevvwR4FeeCNEeAPqo6Ox/jDrUfIacl4RJgdk6bhVR1NDAaIBAI2NJkokJSQhzP921JckIcT3+6lr2HjnD3+Y0QCWsXmzFFJreL1/8b6KSqq0WkHfAEcEY+xp0F1Ax6XAPI6VCKvthmIVMMJcTH8VTvdFKS4hk58zt27z/Mw92aEhdnxcBEjtwKwWFVXQ2gqt+ISJl8jnsB0EBE6gA/4HzZX5m9kYik4hSYq/M5fmOiQlyc8MilzShdIoFRX65n94HDPNmrBQnx4WyZNcZ7uRWCyiJyR06PVfWZ3EasqodFZBAwDYgHXlXVlSIywH39JbdpD2C6qu4p0DswJgqICPd2bUzZEok8OW0Nuw8cZvgVrSiRGO93NGOQnHZgiciDuQ2oqg97kigPgUBAMzKsqyMTvcbN2ciDU1bSsX5FRl8ToFRybr/HjCkcIrJQVQOhXsvxE+jXF70xxd11HWpTKjmBe95dxpVjvmFsvzaUL5XkdywTw2wjpTE+6NW6BiOvOpVVP+2k96i5/LRjn9+RTAyzQmCMT85rWoXx17fl5x376TVyLuu37vY7kolR4fQ+anuzjPHIaXUrMvnm09h/6Ai9X5rLih92+B3JxKBw1ggyReTJ7P0EGWMKR7Pqqbw9oD0lEuPpM2ouszO3+R3JxJhwCkELYC0wRkTmud09lPU4lzExpW5aad79WwdqlC9Jv9fmM9W6sTZFKJwrlO1S1ZdVtQNOZ3MPAj+JyDgRqe95QmNiRJXUErw1oD2tapZnyOTFjJ29we9IJkaEtY9ARLqJyPvA88DTQF1gKvCRx/mMiSmpKYmMv6Et555yEg9N/ZYnPlltndUZz4VzJss6YAbwpKrOCXr+HRHp7E0sY2LXsZ5LH/jvCl6c+R2/7DzAYz2bk2hdUhiPhFMIrlXVr4OfEJGOqjpbVYd4lMuYmBYfJzzaoxlVypbg2c/WsmXXfkZe3ZrSdhay8UA4PzGGhXhueGEHMcb8lYhw2zkNeKJnC+Z8t50+o+ayZdd+v2OZYii36xG0x7lgfVq2zufK4nQiZ4wpApe3qUla2WQGvr6Iy16cw7jr21IvrbTfsUwxktsaQRJQGqdYlAm67QR6eR/NGHPMmY0q8+YtzolnPUfOYf4Gu7S3KTw59j76RwORk1U1Yq7Abb2Pmli2afte+o2dT9av+3jq8nS6pVfzO5KJErn1PprjGoGIPOfeHSEiU7LfPElqjMlVrYolee9vHWhZsxxDJi3mxZmZdnipOWG5HYIwwf37VFEEMcaEp1zJJCbc2Ja73l7GE5+sYfOve/lX92Z2xTNTYLldj2Che7cC8JGqHiiaSMaYvCQnxPNcn5bUqlCSETMy+eH3/Yy4shVlSyT6Hc1EoXB+QnQD1orIBBG5SETsQGZjIkBcnHDn+Y14vGdz5mRuo9fIOWz+da/fsUwUCqevof5AfeBtnIvPfyciY7wOZowJT582tRjnXtegx4uzWbTpN78jmSgT1kZFVT0EfAxMBhYC3b0MZYzJn471K/HewI6USk6g7+h51nupyZdwOp3rKiJjgUyc8wfGAFU9zmWMyaf6lUvz/sCOpNdIZfCkxQz7fJ0dUWTCEs4aQT/gv0BDVb1OVT9S1cPexjLGFESFUkm8fmM7LmtVnWc+Xcttk5ew/9ARv2OZCBfOPoK+wGKgE4CIpIhImXBG7q5NrBGRTBG5N4c2XURkiYisFJEv8xPeGHO85IR4nr48nbu7NmLqsh/pM2ouv+y0PopMzsLZNHQT8A4wyn2qBs4aQl7DxQMvABcATYArsl/uUkTKAS8C3VS1KdA7X+mNMSGJCAO71GfU1a1Zt2U33UfMZnmWXQ/ZhBbOpqFbgY44fQyhquuAymEM1xbIVNX1qnoQZ0dz9p3MVwLvqeomd9xbwg1ujMnbeU2r8O7fOhAfJ/QeNYcPl/3kdyQTgcIpBAfcL3IA3PMIwtkDVR3YHPQ4y30uWEOgvIjMFJGFInJtqBG510nOEJGMrVu3hjFpY8wxp1Qty/8GdaRZtVRufWMRT09fw9GjthPZ/CmcQvCliNwPpIjIuTjnE0wNYzgJ8Vz2T18C0Bq4CDgf+D8RaXjcQKqjVTWgqoG0tLQwJm2MCVapdDITb2pHn0BNhn+Ryc0TMti1/5DfsUyECKcQ3ANsBZYDt+Bcp/iBMIbLAmoGPa4BZD+4OQv4RFX3qOo24CsgPYxxG2PyKTkhnsd6Nufhbk2ZsWYrPV6cw4Zte/yOZSJAroVAROKA5ar6sqr2VtVe7v1w1isXAA1EpI6IJAF9gey9lv4P6CQiCSJSEmgHrCrA+zDGhEFEuK5DbV6/oR3bdx+g+4iv+XKtbW6NdbkWAlU9CiwVkVr5HbF7rsEgYBrOl/tbqrpSRAaIyAC3zSrgE2AZMB8Yo6or8jstY0z+tK9XkSmDTqdauRT6vzafkTO/s5PPYlg4F6b5AmiD80X9x3qkqnbzNlpodmEaYwrP3oOHueudZXy47CcubF6FJ3qlUzrZ+pUsjnK7ME04//GHCzmPMSZClExKYMQVrUivkcpjH69m3S+7GX1tgDqVSvkdzRShPNcI/tJYpBKwPcx9BJ6wNQJjvDE7cxuD3ljE4SPKc31bcvYpJ/kdyRSigl6q8jT3+P73RKSViKwAVgC/iEhXr8IaY/zRsX4lpg4+nZMrleSGcRk8++laO98gRuS2s3gE8CgwCfgCuFFVqwCdgf8UQTZjTBGrUb4k7wzoQM9Ta/D85+voP3YBv+05mPeAJqrlVggSVHW6qr4N/Kyq8wBUdXXRRDPG+KFEYjxP9W7Boz2aM/e77Vw8/GuWbv7d71jGQ7kVgqNB9/dle83WF40pxkSEK9vV4p2/tQeg90tzmfjN93aIaTGVWyFIF5GdIrILaOHeP/a4eRHlM8b4qEWNcnww+HROq1eRf7y/gr+/vZR9B+36BsVNjoVAVeNVtayqllHVBPf+sceJRRnSGOOf8qWSeK1fG4ae04D3F//ApS/MJnPLbr9jmUIU1jWLjTGxLT5OGHpOQ8b2b8vW3QfoNuJr/rfkB79jmUJihcAYE7YzGqbx4ZDTaVqtLLdNXsJ97y23S2EWA1YIjDH5UjU1hTduOo0BZ9Rj0vxN1otpMWCFwBiTb4nxcdx7QWNe7Rfgpx37uGT410xZmr2XeRMtrBAYYwrsrMYn8eGQTjQ8qTRDJi3m3neX2VFFUcgKgTHmhFQvl8Kbt7RnYJd6vJmxmW4jvmb1zzv9jmXywQqBMeaEJcbHcXfXxoy/vi2/7T1E9xGz7QS0KGKFwBhTaDo1SOPj2zrRtk4F/vH+Cm59YxE79tm1kSOdFQJjTKFKK5PMuP5tufeCxkxf+QsXPj+L+Rt+9TuWyYUVAmNMoYuLEwacUY+3B7QnPk7oO3ouz0xfw+EjR/Me2BQ5KwTGGM+0qlWej27rRI9WNRj2RSa9R83l++12zkGksUJgjPFU6eQEnr48neFXtCJzy24ufH4W7y7Msh3JEcQKgTGmSFySXo1PhnamabVU/v72UoZMXsKOvbYjORJYITDGFJnq5VKYdPNp3HleQz5e/hPnP/cVX6/b5nesmOdpIRCRriKyRkQyReTeEK93EZEdIrLEvf3TyzzGGP/FxwmDzmrA+wM7Uio5nqtf+YaHp660zut85FkhEJF44AXgAqAJcIWINAnRdJaqtnRv/8+rPMaYyNK8RiofDulEvw61eW32Ri4e/jUrftjhd6yY5OUaQVsgU1XXq+pBYDLQ3cPpGWOiTInEeB7q1pQJN7Rl1/5DXPrCbF6YkWmHmRYxLwtBdWBz0OMs97ns2ovIUhH5WESaepjHGBOhOjVIY9rQznRtVoUnp62h10tz+W6rXQWtqHhZCCTEc9mPF1sEnKyq6cBw4L8hRyRys4hkiEjG1q1bCzmmMSYSlCuZxIgrT2XYFa3YuH0PFz4/izGz1nP0qB1m6jUvC0EWUDPocQ3gLx2Wq+pOVd3t3v8ISBSRStlHpKqjVTWgqoG0tDQPIxtj/NYtvRrTb+9MpwaVeOTDVfQdPc9OQvOYl4VgAdBAROqISBLQF5gS3EBEqoiIuPfbunm2e5jJGBMFKpcpwcvXBniqdzqrft5J1+dmMWHuRls78IhnhUBVDwODgGnAKuAtVV0pIgNEZIDbrBewQkSWAsOAvmqnGxpjABGhV+saTL+9M4Ha5fm//63k6le+YfOve/2OVuxItH3vBgIBzcjI8DuGMaYIqSqT5m/m0Y9WcVSVu89vxLXtaxMXF2pXpAlFRBaqaiDUa3ZmsTEm4okIV7arxbTbOxOoXYGHpn5Ln9FzWW9HFhUKKwTGmKhRvVwK4/q34ane6az5eRcXPD+LUV9+Z+cdnCArBMaYqHJs38Fnd5xB54Zp/Ofj1fQcOYc1P+/yO1rUskJgjIlKlcuWYPQ1rRl+RSs2/7aPi4fP4pnpa6zPogKwQmCMiVoiwiXp1fj09s5c3KIaw77I5MJhs/hmvR2Fnh9WCIwxUa9i6WSe7dOScde35eDho/QZPY/73lvGjn12vYNwWCEwxhQbZzRMY/rtnbm5c13eXLCZc575kg+X/WRXQ8uDFQJjTLFSMimB+y88hSmDTqdymWRufWMRN43PIOs3OxEtJ1YIjDHFUrPqqfzv1o7848JTmJ25nXOf+YpRX37HITvU9DhWCIwxxVZCfBw3da7Lp3d0pmP9Svzn49VcPOxrMjb+6ne0iGKFwBhT7NUoX5Ix1wUYfU1rdu0/RK+X5nLPO8v4bc9Bv6NFBCsExpiYcV7TKnx6xxnc0rku7yzK4qynZ/JWxuaY79XUCoExJqaUSk7gvgtP4cMhp1MvrTR3v7OMy0fNZeWPsXu9ZCsExpiY1LhKWd66pT2P92zO+m17uGT41/zzfyvYsTf2zj2wQmCMiVlxcUKfNrWY8fcuXHPaybw+73vOfHomby7YFFObi6wQGGNiXmrJRB7u3oypg0+nbqVS3PPucnqMnMOyrN/9jlYkrBAYY4yrabVU3h7Qnmf7pPPj7/vo/sJs7ntvGdt3H/A7mqesEBhjTBARoUerGnzx9zO4oWMd3srIostTMxkzaz0HDxfPk9GsEBhjTAhlSiTywMVNmDa0E61qleeRD1fR9bmvmLF6i9/RCp0VAmOMyUX9ymUY178Nr/ZzLvfbf+wCrnt1Pplbis+FcKwQGGNMHkSEsxqfxCdDO/PARaewaNNvnP/cLB6asrJYHG5qhcAYY8KUlBDHjZ3qMvPOLvRpU5PxczdyxlMzGDdnY1R3ZmeFwBhj8qli6WQe7dGcD4d0oknVsjw4ZSXnPfsV01b+HJXXPvC0EIhIVxFZIyKZInJvLu3aiMgREenlZR5jjClMp1Qty8Qb2/FqvwDxccItExbSZ9Q8lmyOrvMPPCsEIhIPvABcADQBrhCRJjm0exyY5lUWY4zxyh/7D27rxL97NGP9tt1c+sJshkxazOZfo+NiOF6uEbQFMlV1vaoeBCYD3UO0Gwy8CxS/Y7KMMTEjIT6Oq9qdzMy7zmTwWfWZ/u3PnP30lzz60aqI36HsZSGoDmwOepzlPvcHEakO9ABeym1EInKziGSISMbWrVsLPagxxnhYoSYAAA8OSURBVBSW0skJ/P28Rsy4swuXpFfj5Vnr6fzkDEZ/9R37Dx3xO15IXhYCCfFc9r0ozwH3qGquc0dVR6tqQFUDaWlphRbQGGO8UjU1hacvT+eDwaeTXrMcj360mjOfcjq0OxxhRxh5WQiygJpBj2sAP2ZrEwAmi8hGoBfwoohc6mEmY4wpUk2rpTL++ra8cVM7KpctwT3vLqfr87P4ZEXkHGEkXgURkQRgLXA28AOwALhSVVfm0H4s8IGqvpPbeAOBgGZkZBRyWmOM8Z6qMm3lzzwxbQ3rt+6hVa1y3NO1MafVrej5tEVkoaoGQr3m2RqBqh4GBuEcDbQKeEtVV4rIABEZ4NV0jTEmUokIXZtVZfrQzjzeszk//b6fvqPncd2r81nxg39XSPNsjcArtkZgjCku9h86wrg5G3lx5nfs2HeIC5pV4fZzG9LwpDKFPq3c1gisEBhjjM927j/EK7M28MrXG9hz8DDd0qsx9JyG1KlUqtCmYYXAGGOiwG97DjJ61nrGzt7IwSNH6XlqdQaf1YCaFUqe8LitEBhjTBTZuusAI2d+x+vffI+q0qdNTQad2YAqqSUKPE4rBMYYE4V+3rGfETPW8eaCzYgId5/fiBs71S3QuHw5asgYY8yJqZJagkcubc4Xf+/CpS2rUaP8iW8iCiXBk7EaY4wpNDUrlOSJXumejd/WCIwxJsZZITDGmBhnhcAYY2KcFQJjjIlxVgiMMSbGWSEwxpgYZ4XAGGNinBUCY4yJcVHXxYSI7ALW+J2jACoB2/wOUQDRmDsaM0N05o7GzBCbuU9W1ZDX+o3GM4vX5NRfRiQTkQzLXTSiMTNEZ+5ozAyWOzvbNGSMMTHOCoExxsS4aCwEo/0OUECWu+hEY2aIztzRmBks919E3c5iY4wxhSsa1wiMMcYUIisExhgT4yK2EIhIVxFZIyKZInJvDm26iMgSEVkpIl8WdcYQeXLNLCKpIjJVRJa6mfv7kTM7EXlVRLaIyIocXhcRGea+r2UicmpRZwyRKa/MV7lZl4nIHBHx7qoe+ZBX7qB2bUTkiIj0KqpsuQkndwQuj3l9RiJ1eawpIjNEZJWb67YQbQp3mVTViLsB8cB3QF0gCVgKNMnWphzwLVDLfVw5CjLfDzzu3k8DfgWSImB+dwZOBVbk8PqFwMeAAKcB30RB5g5Aeff+BZGQOZzcQZ+lL4CPgF5+Zw5zfkfU8hhm5khdHqsCp7r3ywBrQ3yXFOoyGalrBG2BTFVdr6oHgclA92xtrgTeU9VNAKq6pYgzZhdOZgXKiIgApXE+eIeLNubxVPUrN0tOugPj1TEPKCciVYsmXWh5ZVbVOar6m/twHlCjSILlIYx5DTAYeBfw+zP9hzByR9ryGE7mSF0ef1LVRe79XcAqoHq2ZoW6TEZqIagObA56nMXxM6IhUF5EZorIQhG5tsjShRZO5hHAKcCPwHLgNlU9WjTxTkg47y2S3YDz6yniiUh1oAfwkt9Z8inSlsdwRPzyKCK1gVbAN9leKtRlMlK7mJAQz2U/zjUBaA2cDaQAc0Vknqqu9TpcDsLJfD6wBDgLqAd8KiKzVHWn1+FOUDjvLSKJyJk4heB0v7OE6TngHlU94vxQjRqRtjyGI6KXRxEpjbNmODREpkJdJiN1jSALqBn0uAZO1c7e5hNV3aOq24CvAD93CIaTuT/O6rOqaiawAWhcRPlORDjvLeKISAtgDNBdVbf7nSdMAWCyiGwEegEvisil/kYKS6Qtj+GI2OVRRBJxisBEVX0vRJNCXSYjtRAsABqISB0RSQL6AlOytfkf0ElEEkSkJNAOZ1uaX8LJvAnnFxMichLQCFhfpCkLZgpwrXukwmnADlX9ye9QuRGRWsB7wDUR/qv0L1S1jqrWVtXawDvAQFX9r8+xwhFpy2M4InJ5dPdZvAKsUtVncmhWqMtkRG4aUtXDIjIImIZzBMWrqrpSRAa4r7+kqqtE5BNgGXAUGKOquR6S53dm4F/AWBFZjrNqd4/768lXIjIJ6AJUEpEs4EEgEf7I/RHOUQqZwF6cX1K+CiPzP4GKOL+oAQ5rBPQ2GUbuiJRX7khbHiGseR2RyyPQEbgGWC4iS9zn7gdqgTfLpHUxYYwxMS5SNw0ZY4wpIlYIjDEmxlkhMMaYGGeFwBhjYpwVAmOMiXFWCHLh9v64RERWuL0UlsujfUsRuTDocTfJoefUfOZIFpHP3Cx9sr3WT0SqBT3eKCKVCjid2nn1ilnA8XYRkQ5Bj8eeSK+aIjLE7ZlxYrbns8//h0TkzoJOJx95Orm9RC4RkRSvp1cQInL/iQzj1WfjRAV/3kVkTg5t8vy8ZV+Owpz2pSLSJD/DZBt+qHvOhe+sEORun6q2VNVmOB1S3ZpH+5Y4x/YCoKpTVPWxQsjRCkh0s7yZ7bV+QL4+wD7ogtMbaGEZCFyoqldle/4v878wiUh8Li9fBTzl/n/2hTEuEZGiXvbyXQgKOIxvVPVEPmP9yP9ydClQ4EIADAUiohD42t1qpN+A3UH3BwAvuvfbAnOAxe7fRjhdT28CtuL0X9IH58M1wh3mZOBznBNuPsftrjfb9CoA/3XbzANaAJVxThrZ4Y63XlD7XsBuYI37WgqwEXgYWITTkVZjt20p4FWcM6AX43S7kH36tXG77MU5Ke5Jt/0y4Bb3+S7ATJyzXlcDE/nzfJQL3ee+BoYBH7jj/Bn4wc3YCRjrvj4H50zOkF0tA3cAK9zbUPe5l4CD7nu7PahtqPn/kPueZ7rTGRLU/mpgvtt2FBAfYvobcU5M+xrnTPHzgLnuvH0bp8fKG3F+JGzA6Q4A4K6g+fZw0LxdBbzozv+T82j3MrASmA6kuK/VBz7D6eJ8Ee5nIdR4sr2Px4Aj7nudmNO8zW2YPHLVAz4BFgKzcD9z2cZXGnjN/b8tA3q6z48EMtxxPpxt3of6HFd0p73Y/b99D1QKXl5xTg4bgdMt9ocEdeft/j8XuO97tNs21HLUGvjSfU/TgKrZ3k+HoP/7EnceHDcfcE7aXQB0cYf7D/BvYAh/fo5n+P5d53eASL4FfbDicRb8ru7jskCCe/8c4F33fj/cL/7sj4GpwHXu/euB/4aY3nDgQff+WcAS934X4IMcMs4EAkGPNwKD3fsDcc7wBHgUuNq9Xw6nj/NS2cZVmz8Lwc3AA+79ZHdhreNm2YHTt0kczhfj6UAJnN4Q67jDTDqWGecL+c6g6Yx152cczi+qzBDvq7W7kJTC+RJZCbQKeo+VQgyTff4/hFNskoFKwHacM0tPcf8fiW67F4FrQ4xvI3C3e78STv85pdzH9wD/DHo/x75ozuPPL5g4nGLY2Z23R4HTwmh3GGjptnsr6P/2DdDDvV8C59dkyPHk9FnOa97mMkxuuT4HGrj32wFfhBjX48BzQY+PXSuiQtAyNhNokcfneFjQfL8Ip6O17IXgMuBTd5zVgN+D/j8VgjJMAC7JvhzhfEbmAGnu4z44PQVkf09//N9zmw9AU5wiei5OAUvK7XPsxy0iu5iIICnuKd61car8p+7zqcA4EWmA80FMDGNc7XE+oOB8AJ8I0eZ0oCeAqn4hIhVFJLUAuY91UrUwaJrnAd2CtpmXwDllPaf+YM4DWgRtW00FGuD8ipmvqlkAQfNnN7BeVTe47SfhFJOc/FedLn+/dft5ye504H1V3eNO5z2ctYnFuYwzlA9V9QBwQES2ACfh9C/TGljgdj+RQs59/x/bFHcaTtGa7Q6ThFMEszvPvR3LWRpnvm0Cvlen7/i82m1Q1WNdCywEaotIGaC6qr4PoKr7AUQkp/F8lcs8Kei8DZWrNM6v47eDektNDjHsOThrVbj5j10r4nIRuRnnl3NVnHm8zH0t1Oe487H7qvqhiBwbT7DOwCRVPQL8KCJfBL12pojcjVNEK+AUwanZhm8ENMPpjRScgpJrPz65zQd1upqZ4E6nvTrXK4koVghyt09VW7pfxh/g7CMYhtNHyQxV7SFOf+EzCzDuUH17FFbXsgfcv0f4838sOKvja8Ich+D8Ipv2lydFugSNP3ga+e0zOXgcoYYtrD6Yc8o6TlXvC2P4PUF5PlXVK/JoL8B/VHXUX550Pid7wmyXPXMKOc+PkOMJI2NBhMoVB/yuqi3DmOZfPssiUge4E2ijqr+JyFicHyjZpxf8OSb7eHJwXBsRKYGz9hdQ1c0i8lC26QVnXamq7cOYzjF5zYfmOGsmoX70+M52FodBVXfgbNO7U5zuYVNxtnmDsznimF04l5YLZQ5//iK6Cme7c3Zfua8d+8Ldpnn3jZ7bNINNAwa7PRsiIq3CaP839/0iIg1FpFQu7VcDdd0vMnBWp/ObMdhXwKUiUtKdbg+c7a65CXc6nwO9RKQygIhUEJGT8xhmHtBRROq7w5QUkYYh2k0Drnd/ISIi1Y9Np4DtAHA/B1nidkntHklWMh/jOXTsf0n48zZ4mNxybRCR3u70RUJfH3o6MOjYAxEpj7OJdQ+ww10rvCC3aQVlP7aMXACUz6FNXxGJF+eqXWe6zx/70t/mzq/gI4mCPztrgDQRae9OJ1FEmoaYzh/D5DYfROQynH0bnYFh8ufRhwVZLjxhhSBMqroYZyddX5zNOv8Rkdk4q43HzACaSIjDPHEKSX8RWYbTs+BxF6TG2aYdcNs8BlwXRrSxwEthHLr4L5xNWMvcwwD/lcd4x+DsbFvkth9FLmuQ6hwtMxD4RES+Bn7B2ZcAzipxDzdjpzDeE+pcqm8szg7db3C2Eee16SK3+R887m+BB4Dp7rz+FGezRG55tuIU/UnuMPMI0Xe9qk4H3sC5MMtynJ3qxy3s4bbL5hpgiDv9OUCVfIxnNM7/fmI+5u0fw+SR6yrgBhFZirOpJfslWgEewbmC2Qq33ZmquhRnc9RKnJ36s/OYDjg7kDuLyCKcTWKbQrR5H1iHsx9kJM5OX1T1d5yd3ctxDspYEDTMWNzlCGeZ7gU87mZdQuij3iYDd4nIYhGpF2o+iHNo62PADep0hz4CeN4dfjTwsYjMCON9e8p6HzWFRkRKq+pud63jBWCdqj7rdy5jTO5sjcAUppvcX1QrcTaf5We7tTHGJ7ZGYIwxMc7WCIwxJsZZITDGmBhnhcAYY2KcFQJjjIlxVgiMMSbG/X+BsbR/yqLX1QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "ref_length = np.ones(100)\n",
    "can_length = np.linspace(1.5, 0.5, 100)\n",
    "x = ref_length / can_length\n",
    "y = 1 - x\n",
    "y = np.exp(y)\n",
    "y = np.minimum(np.ones(y.shape), y)\n",
    "\n",
    "# Code for in order to make the plot\n",
    "fig, ax = plt.subplots(1)\n",
    "lines = ax.plot(x, y)\n",
    "ax.set(\n",
    "    xlabel=\"Ratio of the length of the reference to the candidate text\",\n",
    "    ylabel=\"Brevity Penalty\",\n",
    ")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The brevity penalty penalizes generated translations that are too short compared to the closest reference length with an exponential decay. The brevity penalty compensates for the fact that the BLEU score has no recall term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N-Gram Precision (example):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAVVklEQVR4nO3dcdBddX3n8ffHSIaqKIs8bjEhJmKUzW4laozuuLsoLTZAMZV2tkHXWorGbKXK0LKmu611p7OzuJSOtSBpliLabZvqSCWWaHQFa7vW3QQNQqCx2UjlMa4EbaFgCwa++8e90cvNfZ6cPHnOfUjO+zVzJ/ec87v3+eY3mXye3zm/c36pKiRJ3fWUuS5AkjS3DAJJ6jiDQJI6ziCQpI4zCCSp45461wUcrpNPPrkWL14812VI0lHltttuu7+qJkYdO+qCYPHixWzfvn2uy5Cko0qSv5nqmKeGJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeq4VoMgyaoku5LsTrJ+xPFnJflEktuT7ExyUZv1SJIO1loQJJkHXAOcAywDLkyybKjZ24G7quoM4NXAVUnmt1WTJOlgbY4IVgK7q2pPVT0KbAJWD7Up4IQkAZ4BfAfY32JNkqQhbd5ZvAC4d2B7EnjFUJurgc3AXuAE4Geq6vHhL0qyFlgLsGjRohkXtHj9zTP+7LHgnivOm+sSJD0JtTkiyIh9w8uh/TiwA3gusBy4OskzD/pQ1caqWlFVKyYmRj4qQ5I0Q20GwSRw6sD2Qnq/+Q+6CLixenYDXwNOb7EmSdKQNoNgG7A0yZL+BeA19E4DDfo68KMASf4p8CJgT4s1SZKGtHaNoKr2J7kE2ArMA66vqp1J1vWPbwB+A7ghyR30TiW9q6rub6smSdLBWn0MdVVtAbYM7dsw8H4v8No2a5AkTc87iyWp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOazUIkqxKsivJ7iTrRxy/PMmO/uvOJI8lOanNmiRJT9RaECSZB1wDnAMsAy5MsmywTVVdWVXLq2o58CvAn1XVd9qqSZJ0sDZHBCuB3VW1p6oeBTYBq6dpfyHwRy3WI0kaoc0gWADcO7A92d93kCRPA1YBH2uxHknSCG0uXp8R+2qKtucD/2uq00JJ1gJrARYtWjQ71WlGFq+/ea5LmFP3XHHeXJcgzbo2RwSTwKkD2wuBvVO0XcM0p4WqamNVraiqFRMTE7NYoiSpzSDYBixNsiTJfHr/2W8ebpTkWcCZwE0t1iJJmkJrp4aqan+SS4CtwDzg+qramWRd//iGftPXA5+uqofbqkWSNLU2rxFQVVuALUP7Ngxt3wDc0GYdkqSpeWexJHWcQSBJHWcQSFLHGQSS1HEGgSR1nEEgSR1nEEhSxxkEktRxBoEkdZxBIEkdZxBIUscZBJLUcQaBJHWcQSBJHWcQSFLHGQSS1HEGgSR1XKtBkGRVkl1JdidZP0WbVyfZkWRnkj9rsx5J0sFaW6oyyTzgGuBsYBLYlmRzVd010OZE4APAqqr6epLntFWPJGm0NkcEK4HdVbWnqh4FNgGrh9q8Abixqr4OUFX3tViPJGmENhevXwDcO7A9CbxiqM0LgeOSfA44Afjtqvrw8BclWQusBVi0aFErxUrjsHj9zXNdwpy654rz5roEjdDmiCAj9tXQ9lOBlwHnAT8O/FqSFx70oaqNVbWiqlZMTEzMfqWS1GFtjggmgVMHthcCe0e0ub+qHgYeTvJ54Azgqy3WJUka0OaIYBuwNMmSJPOBNcDmoTY3Af86yVOTPI3eqaO7W6xJkjSktRFBVe1PcgmwFZgHXF9VO5Os6x/fUFV3J/kU8BXgceC6qrqzrZokSQdr89QQVbUF2DK0b8PQ9pXAlW3WIUmamncWS1LHHXJE0J/FcznwvMH2VXVWi3VJksakyamhjwIbgP8OPNZuOZKkcWsSBPur6trWK5EkzYkm1wg+keQXkpyS5KQDr9YrkySNRZMRwZv7f14+sK+A589+OZKkcTtkEFTVknEUIkmaG01mDR0H/Hvg3/R3fQ743ar6Xot1SZLGpMmpoWuB4+itGwDwpv6+t7RVlCRpfJoEwcur6oyB7VuS3N5WQZKk8Woya+ixJKcd2EjyfLyfQJKOGU1GBJcDtybZQ2+NgecBF7ValSRpbJrMGvpskqXAi+gFwV9V1SOtVyZJGospgyDJWVV1S5ILhg6dloSqurHl2iRJYzDdiOBM4Bbg/BHHCjAIJOkYMGUQVNWv9//0eoAkHcMOOWsoyTuTPDM91yX5UpLXjqM4SVL7mkwf/fmqehB4LfAcejOGrmjy5UlWJdmVZHeS9SOOvzrJA0l29F/vPqzqJUlHrMn00fT/PBf4YFXdniTTfQAgyTzgGuBsYBLYlmRzVd011PTPq+onDqdoSdLsaTIiuC3Jp+kFwdYkJ9BbaP5QVgK7q2pPVT0KbAJWz7xUSVIbmowILgaWA3uq6rtJnk2zG8oWAPcObE8CrxjR7l/2H1mxF/jlqto53CDJWmAtwKJFixr8aEnHosXrb57rEubUPVec18r3TjkiSHJ6/+3y/p/PT/JShtYunsao00c1tP0l4Hn9Zxn9DvDxUV9UVRurakVVrZiYmGjwoyVJTU33H/pl9H4Lv2rEsQIOtXj9JHDqwPZCer/1/+BLehehD7zfkuQDSU6uqvsP8d2SpFky3X0Ea/t/vmaG370NWJpkCfANYA3whsEGSX4Y+FZVVZKV9EYo357hz5MkzUCT+wjenuTEge1/kuQXDvW5qtoPXAJsBe4GPlJVO5OsS7Ku3+yngTv71wjeD6ypquHTR5KkFjU51//WqrrmwEZV/W2St/KDhWqmVFVbgC1D+zYMvL8auLp5uZKk2dZk+uhTBu8b6N8fML+9kiRJ49RkRLAV+EiSDfQuEq8DPtVqVZKksWkSBO8C3kZvAfsAnwaua7MoSdL4NFmY5vEkNwC3VNWu9kuSJI1Tk1lDrwN20D8dlGR5ks1tFyZJGo8mF4t/nd5zg/4OoKp2AItbrEmSNEZNgmB/VT3QeiWSpDnR5GLxnUneAMzrL2L/DuAL7ZYlSRqXJiOCXwT+OfAI8IfAA8ClbRYlSRqfaUcE/ZvHNlfVjwH/aTwlSZLGadoRQVU9Bnw3ybPGVI8kacyaXCP4R+COJJ8BHj6ws6re0VpVkqSxaRIEN/dfkqRjUJM7iz+UZD5wOr1nDe3qr0EsSToGHDIIkpwL/C7wf+k9a2hJkrdV1SfbLk6S1L4mp4Z+C3hNVe0GSHIavVNFBoEkHQOa3Edw34EQ6NsD3NdSPZKkMWsSBDuTbEnyc0neDHwC2JbkgiQXTPfBJKuS7EqyO8n6adq9PMljSX76MOuXJB2hJqeGjge+BZzZ394HnAScT+/i8Y2jPtS/Ge0a4Gxgkl54bK6qu0a0ey+9BXAkSWPWZNbQRTP87pXA7qraA5BkE7AauGuo3S8CHwNePsOfI0k6Ak1ODc3UAuDege3J/r7vS7IAeD2wgWkkWZtke5Lt+/btm/VCJanL2gyCjNhXQ9vvA97Vf5TFlKpqY1WtqKoVExMTs1agJKnZNYKZmgROHdheCOwdarMC2JQE4GTg3CT7q+rjLdYlSRrQ5IayE4Gfpbcq2ffbN3jW0DZgaZIlwDeANcAbBhtU1ZKBn3MD8KeGgCSNV5MRwRbgi8AdwONNv7iq9ie5hN5soHnA9VW1M8m6/vFprwtIksaj0fTRqrpsJl9eVVvoBcngvpEBUFU/N5OfIUk6Mk0uFv9+krcmOSXJSQderVcmSRqLJiOCR4Er6a1QdmDWTwHPb6soSdL4NAmCy4AXVNX9bRcjSRq/Rs8aAr7bdiGSpLnRZETwGLAjya3AIwd2ulSlJB0bmgTBx/svSdIxqNFSleMoRJI0N5rcWbwU+K/AMnqPpAagqpw1JEnHgCYXiz8IXAvsB14DfBj4/TaLkiSNT5Mg+KGq+iyQqvqbqnoPcFa7ZUmSxqXJxeJ/TPIU4K/7zw76BvCcdsuSJI1LkxHBpcDTgHcALwP+HfDmNouSJI3PtCOC/nrC/7aqLgceAma6bKUk6Ulq2hFBf+Wwl6W/cowk6djT5BrBl4GbknwUePjAzqq6sbWqJElj0yQITgK+zRNnChVgEEjSMaDJncVeF5CkY1iTO4vfP2L3A8D2qrrpEJ9dBfw2vaUqr6uqK4aOrwZ+g94SmPuBS6vqLxrWLkmaBU2mjx4PLAf+uv96Mb3TRRcned9UH+rPOLoGOIfe4ykuTLJsqNlngTOqajnw88B1h/03kCQdkSbXCF4AnFVV+wGSXAt8Gjib3oL2U1kJ7K6qPf3PbQJWA3cdaFBVDw20fzo/WAFNkjQmTUYEC+j9J33A04Hn9qeWPjL6I9//3L0D25P9fU+Q5PVJ/gq4md6o4CBJ1ibZnmT7vn37GpQsSWqqSRD8N3oL03wwyQ30ppP+ZpKnA/9zms+NuvfgoN/4q+pPqup04CfpXS84+ENVG6tqRVWtmJiYaFCyJKmpJrOGfi/JFnqnegL8x6ra2z98+TQfnQROHdheCOydoi1V9fkkpyU52fWRJWl8mowIqKpv9mcILR8IgUPZBixNsiTJfGANsHmwQZIXHLhrOclLgfn07lmQJI1JoyAY8LqmDfsXly8BtgJ3Ax+pqp1J1iVZ12/2U8CdSXbQm2H0M1XlBWNJGqMms4YGHdYzh6pqC7BlaN+GgffvBd57mDVIkmbR4Y4IXtZKFZKkOTPliCDJu6fYD1BVNXKGjyTp6DLdqaGHR+x7OnAx8GymmOopSTq6TBkEVXXVgfdJTgDeSW9hmk3AVVN9TpJ0dDnUCmUnAZcBbwQ+BLy0qv52HIVJksZjumsEVwIXABuBHxl6LpAk6Rgx3ayhXwKeC/wqsDfJg/3X3yd5cDzlSZLaNt01gsOdWipJOgr5n70kdZxBIEkdZxBIUscZBJLUcQaBJHWcQSBJHWcQSFLHGQSS1HEGgSR1XKtBkGRVkl1JdidZP+L4G5N8pf/6QpIz2qxHknSw1oIgyTx66xCfAywDLkyybKjZ14Azq+rF9NY32NhWPZKk0docEawEdlfVnqp6lN46BqsHG1TVFwYea/1FYGGL9UiSRmgzCBYA9w5sT/b3TeVi4JOjDiRZm2R7ku379u2bxRIlSW0GQUbsq5ENk9fQC4J3jTpeVRurakVVrZiYmJjFEiVJ065QdoQmgVMHthcCe4cbJXkxcB1wTlV9u8V6JEkjtDki2AYsTbIkyXxgDbB5sEGSRcCNwJuq6qst1iJJmkJrI4Kq2p/kEmArMA+4vqp2JlnXP74BeDfwbOADSQD2V9WKtmqSJB2szVNDVNUWYMvQvg0D798CvKXNGiRJ0/POYknqOINAkjrOIJCkjjMIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4wwCSeo4g0CSOs4gkKSOMwgkqeMMAknqOINAkjrOIJCkjjMIJKnjDAJJ6rhWgyDJqiS7kuxOsn7E8dOT/GWSR5L8cpu1SJJGa22pyiTzgGuAs4FJYFuSzVV110Cz7wDvAH6yrTokSdNrc0SwEthdVXuq6lFgE7B6sEFV3VdV24DvtViHJGkabQbBAuDege3J/r7DlmRtku1Jtu/bt29WipMk9bQZBBmxr2byRVW1sapWVNWKiYmJIyxLkjSozSCYBE4d2F4I7G3x50mSZqDNINgGLE2yJMl8YA2wucWfJ0magdZmDVXV/iSXAFuBecD1VbUzybr+8Q1JfhjYDjwTeDzJpcCyqnqwrbokSU/UWhAAVNUWYMvQvg0D7/8fvVNGkqQ54p3FktRxBoEkdZxBIEkdZxBIUscZBJLUcQaBJHWcQSBJHWcQSFLHGQSS1HEGgSR1nEEgSR1nEEhSxxkEktRxBoEkdZxBIEkdZxBIUscZBJLUca0GQZJVSXYl2Z1k/YjjSfL+/vGvJHlpm/VIkg7WWhAkmQdcA5wDLAMuTLJsqNk5wNL+ay1wbVv1SJJGa3NEsBLYXVV7qupRYBOweqjNauDD1fNF4MQkp7RYkyRpSJuL1y8A7h3YngRe0aDNAuCbg42SrKU3YgB4KMmu2S11bE4G7p+rH573ztVPnlX24ZGx/47M0dx/z5vqQJtBkBH7agZtqKqNwMbZKGouJdleVSvmuo6jmX14ZOy/I3Os9l+bp4YmgVMHthcCe2fQRpLUojaDYBuwNMmSJPOBNcDmoTabgZ/tzx56JfBAVX1z+IskSe1p7dRQVe1PcgmwFZgHXF9VO5Os6x/fAGwBzgV2A98FLmqrnieJo/701pOAfXhk7L8jc0z2X6oOOiUvSeoQ7yyWpI4zCCSp4wyChpJcn+S+JHfOdS1HoySnJrk1yd1JdiZ551zXdDRJcnyS/5Pk9n7//ee5rulolGReki8n+dO5ruXJxCBo7gZg1ZF+SZI27914MtsP/FJV/TPglcDbRzxypJGO9uEjwFlVdQawHFjVn2l32Drafwe8E7j7SL7gWOy/Y+4v1Jaq+nySxdO1SfJrwBvp3S19P3BbVf1mks8BXwBeBWxO8lXgV4H5wLeBN1bVt5K8B1gCnAK8ELiM3n+a5wDfAM6vqu/N+l9uDPrTgr/Zf//3Se6mdxf5XYPt7MPRqjer46H+5nH910EzPey/qSVZCJwH/Bd6f69RbTrZf44IZkmSFcBPAS8BLgCG7z48sarOrKqrgL8AXllVL6H3DKb/MNDuNHr/WFcD/wO4tap+BPiH/v6jXj9QXwL876H99uE0+qc1dgD3AZ+pKvvv8LyP3t/z8VEHu9x/jghmz78CbqqqfwBI8omh43888H4h8Mf9B+zNB742cOyTVfW9JHfQu//iU/39dwCL2yh8nJI8A/gYcGlVPTh02D6cRlU9BixPciLwJ0n+RVUNXrOy/6aQ5CeA+6rqtiSvnqJZZ/vPEcEM9S9+7ui/1jH6uUmDHh54/zvA1f3fEt4GHD9w7BGAqnoc+F794EaPxznKgzvJcfRC4A+q6kb7cGaq6u+AzwGvt/8aexXwuiT30PsN/qwkn7X/egyCGaqqe6tqef+1gd5Q8fz+7I5nMP0Q8Fn0zhcCvLntWp8MkgT4PeDuqvotsA8PR5KJ/kiAJD8E/BjwZfuvmar6lapaWFWL6T3u5paq+lH7r8cgaCjJHwF/CbwoyWSSiwePV9U2es9Ouh24EdgOPDDF170H+GiSP2cOH2k7Zq8C3kTvN7EDv4WdO9jAPpzWKcCtSb5C7zlen6mqJ0yBtP+OTJf7z0dMzKIkz6iqh5I8Dfg8sLaqvjTXdR1N7MMjY/8dma7235PyfNVRbGN/bvzxwIe68A+oBfbhkbH/jkwn+88RgSR1nNcIJKnjDAJJ6jiDQJI6ziCQpI4zCCSp4/4/8j88aniU30cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = {\"1-gram\": 0.8, \"2-gram\": 0.7, \"3-gram\": 0.6, \"4-gram\": 0.5}\n",
    "names = list(data.keys())\n",
    "values = list(data.values())\n",
    "\n",
    "fig, ax = plt.subplots(1)\n",
    "bars = ax.bar(names, values)\n",
    "ax.set(ylabel=\"N-gram precision\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The n-gram precision counts how many unigrams, bigrams, trigrams, and four-grams (i=1,...,4) match their n-gram counterpart in the reference translations. This term acts as a precision metric. Unigrams account for adequacy while longer n-grams account for fluency of the translation. To avoid overcounting, the n-gram counts are clipped to the maximal n-gram count occurring in the reference ($m_{n}^{ref}$). Typically precision shows exponential decay with the with the degree of the n-gram."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### N-gram BLEU score (example):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAD4CAYAAADhNOGaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAXeElEQVR4nO3df5xddX3n8debkSwF+VFkrJgfJGCQTVVQx2AfukWpaAAxldo2gEgRTdOK4OKyxl20dn3sti7Fh3VBpilNsbprWh9GiXUAXX48bGttkyi/Ao2djdaMoRIE+aUlTHjvH+dMe7m5c+dkZs4dZs77+Xjcx73nnO8593PPI4985vs93x+yTURENNcBMx1ARETMrCSCiIiGSyKIiGi4JIKIiIZLIoiIaLjnzHQA++uoo47y4sWLZzqMiIhZZevWrQ/a7u90bNYlgsWLF7Nly5aZDiMiYlaR9E/jHUvTUEREwyURREQ0XBJBRETDJRFERDRcEkFERMMlEURENFytiUDSCknbJQ1LWtvh+OGSvizpTknbJF1YZzwREbGv2hKBpD7gGuB0YBlwjqRlbcXeA9xr+0TgdcBVkubVFVNEROyrzhrBcmDY9g7be4ANwMq2MgYOlSTgucBDwGiNMUVERJs6RxbPB3a2bI8AJ7eVuRrYBOwCDgV+3fbT7ReStBpYDbBo0aJJB7R47Vcmfe5c8L3fP3OmQ4iIZ6E6awTqsK99ObQ3AXcALwROAq6WdNg+J9nrbA/YHujv7zhVRkRETFKdiWAEWNiyvYDiL/9WFwIbXRgGvgucUGNMERHRps5EsBlYKmlJ+QB4FUUzUKvvA78EIOnngBcDO2qMKSIi2tT2jMD2qKSLgZuBPmC97W2S1pTHB4GPAtdLupuiKekDth+sK6aIiNhXrdNQ2x4Chtr2DbZ83gW8sc4YIiKiu4wsjohouFm3ME3MrHTBTRfcmHtSI4iIaLgkgoiIhksiiIhouCSCiIiGSyKIiGi4JIKIiIZLIoiIaLiMI4jooYzDyDiMZ6PUCCIiGi6JICKi4ZIIIiIaLokgIqLhkggiIhouiSAiouFqTQSSVkjaLmlY0toOxy+XdEf5ukfSXklH1hlTREQ8U23jCCT1AdcAp1EsZL9Z0ibb946VsX0lcGVZ/izgP9p+qK6YImJ2yziMesZh1FkjWA4M295hew+wAVjZpfw5wOdqjCciIjqoMxHMB3a2bI+U+/Yh6WBgBfCFGuOJiIgO6kwE6rDP45Q9C/ib8ZqFJK2WtEXSlt27d09bgBERUW8iGAEWtmwvAHaNU3YVXZqFbK+zPWB7oL+/fxpDjIiIOhPBZmCppCWS5lH8Z7+pvZCkw4FTgBtqjCUiIsZRW68h26OSLgZuBvqA9ba3SVpTHh8si74V+KrtJ+qKJSIixlfrNNS2h4Chtn2DbdvXA9fXGUdERIwvI4sjIhouiSAiouGSCCIiGi6JICKi4SZ8WCzpeOBy4JjW8rZPrTGuiIjokSq9hj4PDAJ/DOytN5yIiOi1Kolg1Pa1tUcSEREzosozgi9L+m1JR0s6cuxVe2QREdETVWoEF5Tvl7fsM3Ds9IcTERG9NmEisL2kF4FERMTMqNJr6EDgt4BfLHfdDvyR7adqjCsiInqkStPQtcCBwKfK7fPLfe+qK6iIiOidKongVbZPbNm+VdKddQUUERG9VaXX0F5Jx41tSDqWjCeIiJgzqtQILgduk7SDYvnJY4ALa40qIiJ6pkqvoVskLQVeTJEI/sH2k7VHFhERPTFuIpB0qu1bJZ3ddug4SdjeWHNsERHRA91qBKcAtwJndThmYMJEIGkF8IcUS1VeZ/v3O5R5HfAJip5JD9o+ZeKwIyJiuoybCGz/Tvk+qecBkvqAa4DTgBFgs6RNtu9tKXMERbfUFba/L+n5k/muiIiYvAl7DUm6VNJhKlwn6VuS3ljh2suBYds7bO8BNgAr28qcC2y0/X0A2w/s7w+IiIipqdJ99J22HwXeCDyfosfQPk08HcwHdrZsj5T7Wh0P/Kyk2yVtlfSOTheStFrSFklbdu/eXeGrIyKiqiqJQOX7GcCf2r6zZV+V81q5bfs5wCuBM4E3AR8qF8J55kn2OtsDtgf6+/srfHVERFRVZRzBVklfBZYAH5R0KPB0hfNGgIUt2wuAXR3KPGj7CeAJSV8HTgS+U+H6ERExDarUCC4C1lJMNfETYB7VBpRtBpZKWiJpHrAK2NRW5gbgP0h6jqSDgZOB+ypHHxERU9ZtHMEJtv8BOKncdaxUpUWoYHtU0sXAzRTdR9fb3iZpTXl80PZ9km4C7qKoZVxn+55J/paIiJiEbk1DlwGrgas6HDMw4eL1toeAobZ9g23bVwJXThhpRETUots4gtXl++t7F05ERPRalXEE7ykHfo1t/6yk3643rIiI6JUqD4vfbfvHYxu2HwbeXV9IERHRS1USwQFqeUpcTh0xr76QIiKil6qMI7gZ+AtJgxQPidcAN9UaVURE9EyVRPAB4DcpFrAX8FXgujqDioiI3qmyMM3Tkq4HbrW9vf6QIiKil6r0GnoLcAdlc5CkkyS1jxCOiIhZqsrD4t+hmFL6xwC27wAW1xhTRET0UJVEMGr7kdojiYiIGVHlYfE9ks4F+spF7C8BvlFvWBER0StVagTvBX4eeBL4P8AjwPvqDCoiInqna42gHDy2yfYbgP/am5AiIqKXutYIbO8FfiLp8B7FExERPVblGcG/AHdL+hrwxNhO25fUFlVERPRMlUTwlfIVERFzUJWRxZ8ul5o8gWKuoe2299QeWURE9ESVkcVnAP8P+CRwNTAs6fQqF5e0QtJ2ScOS1nY4/jpJj0i6o3x9eH9/QERETE2VpqGPA6+3PQwg6TiKpqIbu51U9ji6BjgNGAE2S9pk+962on9l+837HXlEREyLKuMIHhhLAqUdwAMVzlsODNveUTYlbQBWTiLGiIioUZVEsE3SkKTfkHQB8GWKv+7PlnR2l/PmAztbtkfKfe1+QdKdkm6U9POdLiRptaQtkrbs3r27QsgREVFVlaahg4AfAqeU27uBI4GzKB4ebxznPHXY57btbwHH2H68fBbxJWDpPifZ64B1AAMDA+3XiIiIKajSa+jCSV57BFjYsr0A2NV27UdbPg9J+pSko2w/OMnvjIiI/VSlaWiyNgNLJS0pu5+uAp6xjoGkF4ythyxpeRnPj2qMKSIi2lRpGpoU26OSLqZY87gPWG97m6Q15fFB4G3Ab0kaBX4KrLKdpp+IiB6qLRFA0dwDDLXtG2z5fDXF2ISIiJghEyYCSUcA76BYlexfy2euoYiIuaFKjWAI+CZwN/B0veFERESvVeo+avuy2iOJiIgZUaXX0GckvVvS0ZKOHHvVHllERPRElRrBHuBKihXKxnr0GDi2rqAiIqJ3qiSCy4AXZZBXRMTcVGmuIeAndQcSEREzo0qNYC9wh6TbgCfHdqb7aETE3FAlEXypfEVExBxUaanKXgQSEREzo8rI4qXA7wHLKKakBsB2eg1FRMwBVR4W/ylwLTAKvB74M+AzdQYVERG9UyUR/IztWwDZ/ifbHwFOrTesiIjolSoPi/9F0gHAP5bTSv8AeH69YUVERK9UqRG8DzgYuAR4JfB24II6g4qIiN7pWiOQ1Af8mu3LgceByS5bGRERz1JdawS29wKvHFtOMiIi5p4qTUPfBm6QdL6ks8deVS4uaYWk7ZKGJa3tUu5VkvZKelvVwCMiYnpUeVh8JMWC8q09hQxs7HZS2ax0DXAaMAJslrTJ9r0dyn2MYm3jiIjosSojiyf7XGA5MGx7B4CkDcBK4N62cu8FvgC8apLfExERU1BlZPEnO+x+BNhi+4Yup84HdrZsjwAnt117PvBWitrGuIlA0mpgNcCiRYsmCjkiIvZDlWcEBwEnAf9Yvl5G0Vx0kaRPdDmv0wNmt21/AvhA+VB6XLbX2R6wPdDf318h5IiIqKrKM4IXAafaHgWQdC3wVYq2/7u7nDcCLGzZXgDsaiszAGwoOyUdBZwhadR2ZjuNiOiRKolgPnAIRXMQ5ecX2t4r6cnxT2MzsFTSEorRyKuAc1sL2F4y9lnS9cBfJglERPRWlUTwPykWprmdornnF4H/IekQ4P+Od5Lt0XJKipuBPmC97W2S1pTHB6cafERETF2VXkN/ImmIoheQgP9ie6yJ5/IJzh0Chtr2dUwAtn+jSsARETG9qjwsxvb9ZQ+hk1qSQEREzAGVEkGLt9QSRUREzJj9TQSZcygiYo7Z30TwylqiiIiIGTPuw2JJHx5nP4Btf7SuoCIione69Rp6osO+Q4CLgOcBSQQREXPAuInA9lVjnyUdClxKsTDNBuCq8c6LiIjZZaIVyo4ELgPOAz4NvML2w70ILCIieqPbM4IrgbOBdcBLbT/es6giIqJnuvUaej/wQuAKYJekR8vXY5Ie7U14ERFRt27PCPa3a2lERMxC+c8+IqLhkggiIhouiSAiouGSCCIiGq5b99HH2HeN4X9l+7BaIoqIiJ7q1mvoUABJ/w34Z+AzFLOPngcc2pPoIiKidlWaht5k+1O2H7P9qO1rgV+pcnFJKyRtlzQsaW2H4ysl3SXpDklbJL12f39ARERMTZVEsFfSeZL6JB0g6Txg70QnSeoDrgFOB5YB50ha1lbsFuBE2ycB7wSu27/wIyJiqqokgnOBXwN+WL5+tdw3keXAsO0dtvdQTFa3srWA7cdtjz2HOIQuzyQiIqIeVRav/x5t/4FXNB/Y2bI9ApzcXkjSW4HfA54PnNnpQpJWA6sBFi1aNIlQIiJiPBPWCCQdL+kWSfeU2y+TdEWFa3da1nKfv/htf9H2CcAvM84aB7bX2R6wPdDf31/hqyMioqoqTUN/DHwQeArA9l3AqgrnjQALW7YXALvGK2z768Bxko6qcO2IiJgmVRLBwbb/vm3faIXzNgNLJS2RNI8ieWxqLSDpRSrXvpT0CmAe8KMK146IiGky4TMC4EFJx1E260h6G3D/RCfZHpV0MXAz0Aest71N0pry+CBFN9R3SHoK+Cnw6y0PjyMiogeqJIL3UCxOc4KkHwDfBd5e5eK2h4Chtn2DLZ8/BnyscrQRETHtqvQa2gG8QdIhwAG2H6s/rIiI6JVucw293fZnJV3Wth8A2x+vObaIiOiBbjWCg8v3zCsUETGHdUsEx5Xv99r+fC+CiYiI3uvWffQMSQdSjCGIiIg5qluN4CbgQeAQSY+27BfgrEcQETE3jFsjsH257cOBr9g+rOV1aJJARMTcMeHIYtuTmXAuIiJmiXETgaS/Lt8fk/Ro+3vvQoyIiDp1W6ryteV7uo9GRMxh3QaUHdntRNsPTX84ERHRa916DW2lmGhOwCLg4fLzEcD3gSW1RxcREbXr1mtoie1jKWYPPcv2UbafB7wZ2NirACMiol5V1iN4VTmLKAC2bwROqS+kiIjoparrEVwBfJaiqejtZPGYiIg5o0qN4BygH/gi8CWKRebPqTOoiIjonSrrETwEXCrpMOBp24/XH1ZERPTKhDUCSS+V9G3gbmCbpK2SXlLl4pJWSNouaVjS2g7Hz5N0V/n6hqQT9/8nRETEVFRpGvoj4DLbx9g+Bng/xdKVXUnqA64BTgeWAedIWtZW7LvAKbZfBny0ynUjImJ6VUkEh9i+bWzD9u3AIRXOWw4M295hew+wAXjGvEW2v2H74XLzm8CCSlFHRMS0qZIIdkj6kKTF5esKir/kJzIf2NmyPVLuG89FwI2dDkhaLWmLpC27d++u8NUREVFVlUTwTopeQxspeg71AxdWOE8d9rljQen1FIngA52O215ne8D2QH9/f4WvjoiIqqr0GnoYuGQS1x4BFrZsLwB2tReS9DLgOuB02xmfEBHRY90mndvU7UTbb5ng2puBpZKWAD8AVgHntn3HIoqaxvm2v1Mp4oiImFbdagS/QNHG/zng7+jc1DMu26OSLqaYq6gPWG97m6Q15fFB4MPA84BPSQIYtT2w378iIiImrVsieAFwGsUo4nOBrwCfs72t6sXLOYqG2vYNtnx+F/Cu/Qk4IiKmV7fZR/favsn2BcCrgWHgdknv7Vl0ERFRu64PiyX9O+BMilrBYuCTZArqiIg5pdvD4k8DL6Ho2/+7tu/pWVQREdEz3WoE5wNPAMcDl5QPc6F4aGzbh9UcW0RE9EC3xeurDDaLiIhZLv/ZR0Q0XBJBRETDJRFERDRcEkFERMMlEURENFwSQUREwyURREQ0XBJBRETDJRFERDRcEkFERMMlEURENFwSQUREw9WaCCStkLRd0rCktR2OnyDpbyU9Kek/1RlLRER01nVhmqmQ1AdcQ7Hc5QiwWdIm2/e2FHsIuAT45briiIiI7uqsESwHhm3vsL0H2ACsbC1g+wHbm4GnaowjIiK6qDMRzAd2tmyPlPv2m6TVkrZI2rJ79+5pCS4iIgp1JgJ12OfJXMj2OtsDtgf6+/unGFZERLSqMxGMAAtbthcAu2r8voiImIQ6E8FmYKmkJZLmAauATTV+X0RETEJtvYZsj0q6GLgZ6APW294maU15fFDSC4AtwGHA05LeByyz/WhdcUVExDPVlggAbA8BQ237Bls+/zNFk1FERMyQjCyOiGi4JIKIiIZLIoiIaLgkgoiIhksiiIhouCSCiIiGSyKIiGi4JIKIiIZLIoiIaLgkgoiIhksiiIhouCSCiIiGSyKIiGi4JIKIiIZLIoiIaLgkgoiIhksiiIhouFoTgaQVkrZLGpa0tsNxSfpkefwuSa+oM56IiNhXbYlAUh9wDXA6sAw4R9KytmKnA0vL12rg2rriiYiIzuqsESwHhm3vsL0H2ACsbCuzEvgzF74JHCHp6BpjioiINnUuXj8f2NmyPQKcXKHMfOD+1kKSVlPUGAAel7R9ekPtmaOAB2fqy/WxmfrmaZV7ODW5f1Mzm+/fMeMdqDMRqMM+T6IMttcB66YjqJkkaYvtgZmOYzbLPZya3L+pmav3r86moRFgYcv2AmDXJMpERESN6kwEm4GlkpZImgesAja1ldkEvKPsPfRq4BHb97dfKCIi6lNb05DtUUkXAzcDfcB629skrSmPDwJDwBnAMPAT4MK64nmWmPXNW88CuYdTk/s3NXPy/snep0k+IiIaJCOLIyIaLokgIqLhkggqkrRe0gOS7pnpWGYjSQsl3SbpPknbJF060zHNJpIOkvT3ku4s79/vznRMs5GkPknflvSXMx3Ls0kSQXXXAyumehFJdY7deDYbBd5v+98Drwbe02HKkUoaeg+fBE61fSJwErCi7Gm33xp6/8ZcCtw3lQvMxfs3535QXWx/XdLibmUkfQg4j2K09IPAVtt/IOl24BvAa4BNkr4DXAHMA34EnGf7h5I+AiwBjgaOBy6j+E/zdOAHwFm2n5r2H9cDZbfg+8vPj0m6j2IU+b2t5XIPO3PRq+PxcvPA8rVPT4/cv/FJWgCcCfx3it/VqUwj719qBNNE0gDwK8DLgbOB9tGHR9g+xfZVwF8Dr7b9coo5mP5zS7njKP6xrgQ+C9xm+6XAT8v9s16ZUF8O/F3b/tzDLspmjTuAB4Cv2c792z+foPidT3c62OT7lxrB9HktcIPtnwJI+nLb8T9v+bwA+PNygr15wHdbjt1o+ylJd1OMv7ip3H83sLiOwHtJ0nOBLwDvs/1o2+Hcwy5s7wVOknQE8EVJL7Hd+swq928ckt4MPGB7q6TXjVOssfcvNYJJKh9+3lG+1tB53qRWT7R8/l/A1eVfCb8JHNRy7EkA208DT/nfBno8zSxP3JIOpEgC/9v2xtzDybH9Y+B24K25f5W9BniLpO9R/AV/qqRbcv8KSQSTZHun7ZPK1yBFVfGssnfHc+leBTycor0Q4IK6Y302kCTgT4D7bH8ccg/3h6T+siaApJ8B3gB8O/evGtsftL3A9mKK6W5utf1LuX+FJIKKJH0O+FvgxZJGJF3Uetz2Zoq5k+4ENgJbgEfGudxHgM9L+itmcErbHnsNcD7FX2Jjf4Wd0Vog97Cro4HbJN1FMY/X12w/owtk7t/UNPn+ZYqJaSTpubYfl3Qw8HVgte1vzXRcs0nu4dTk/k1NU+/fs7K9ahZbV/aNPwj4dBP+AdUg93Bqcv+mppH3LzWCiIiGyzOCiIiGSyKIiGi4JIKIiIZLIoiIaLgkgoiIhvv/1AAh2cZ9HsgAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "data = {\"1-gram\": 0.8, \"2-gram\": 0.77, \"3-gram\": 0.74, \"4-gram\": 0.71}\n",
    "names = list(data.keys())\n",
    "values = list(data.values())\n",
    "\n",
    "fig, ax = plt.subplots(1)\n",
    "bars = ax.bar(names, values)\n",
    "ax.set(ylabel=\"Modified N-gram precision\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the n-gram precision is multiplied by the BP, then the exponential decay of n-grams is almost fully compensated. The BLEU score corresponds to a geometric average of this modified n-gram precision."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4 Example Calculations of the BLEU score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we will have a reference translation and 2 candidates translations. We will tokenize all sentences using the NLTK package introduced in Course 2 of this NLP specialization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The NASA Opportunity rover is battling a massive dust storm on planet Mars. -> ['the', 'nasa', 'opportunity', 'rover', 'is', 'battling', 'a', 'massive', 'dust', 'storm', 'on', 'planet', 'mars', '.']\n",
      "\n",
      "\n",
      "The Opportunity rover is combating a big sandstorm on planet Mars. -> ['the', 'opportunity', 'rover', 'is', 'combating', 'a', 'big', 'sandstorm', 'on', 'planet', 'mars', '.']\n",
      "\n",
      "\n",
      "A NASA rover is fighting a massive storm on planet Mars. -> ['a', 'nasa', 'rover', 'is', 'fighting', 'a', 'massive', 'storm', 'on', 'planet', 'mars', '.']\n"
     ]
    }
   ],
   "source": [
    "reference = \"The NASA Opportunity rover is battling a massive dust storm on planet Mars.\"\n",
    "candidate_1 = \"The Opportunity rover is combating a big sandstorm on planet Mars.\"\n",
    "candidate_2 = \"A NASA rover is fighting a massive storm on planet Mars.\"\n",
    "\n",
    "tokenized_ref = nltk.word_tokenize(reference.lower())\n",
    "tokenized_cand_1 = nltk.word_tokenize(candidate_1.lower())\n",
    "tokenized_cand_2 = nltk.word_tokenize(candidate_2.lower())\n",
    "\n",
    "print(f\"{reference} -> {tokenized_ref}\")\n",
    "print(\"\\n\")\n",
    "print(f\"{candidate_1} -> {tokenized_cand_1}\")\n",
    "print(\"\\n\")\n",
    "print(f\"{candidate_2} -> {tokenized_cand_2}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 1: Computing the Brevity Penalty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def brevity_penalty(reference, candidate):\n",
    "    ref_length = len(reference)\n",
    "    can_length = len(candidate)\n",
    "\n",
    "    # Brevity Penalty\n",
    "    if ref_length > can_length:\n",
    "        BP = 1\n",
    "    else:\n",
    "        penalty = 1 - (ref_length / can_length)\n",
    "        BP = np.exp(penalty)\n",
    "\n",
    "    return BP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 2: Computing the Precision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clipped_precision(reference, candidate):\n",
    "    \"\"\"\n",
    "    Bleu score function given a original and a machine translated sentences\n",
    "    \"\"\"\n",
    "\n",
    "    clipped_precision_score = []\n",
    "\n",
    "    for i in range(1, 5):\n",
    "        candidate_n_gram = Counter(\n",
    "            ngrams(candidate, i)\n",
    "        )  # counts of n-gram n=1...4 tokens for the candidate\n",
    "        reference_n_gram = Counter(\n",
    "            ngrams(reference, i)\n",
    "        )  # counts of n-gram n=1...4 tokens for the reference\n",
    "\n",
    "        c = sum(\n",
    "            reference_n_gram.values()\n",
    "        )  # sum of the values of the reference the denominator in the precision formula\n",
    "\n",
    "        for j in reference_n_gram:  # for every n_gram token in the reference\n",
    "            if j in candidate_n_gram:  # check if it is in the candidate n-gram\n",
    "\n",
    "                if (\n",
    "                    reference_n_gram[j] > candidate_n_gram[j]\n",
    "                ):  # if the count of the reference n-gram is bigger\n",
    "                    # than the corresponding count in the candidate n-gram\n",
    "                    reference_n_gram[j] = candidate_n_gram[\n",
    "                        j\n",
    "                    ]  # then set the count of the reference n-gram to be equal\n",
    "                    # to the count of the candidate n-gram\n",
    "            else:\n",
    "\n",
    "                reference_n_gram[j] = 0  # else reference n-gram = 0\n",
    "\n",
    "        clipped_precision_score.append(sum(reference_n_gram.values()) / c)\n",
    "\n",
    "    weights = [0.25] * 4\n",
    "\n",
    "    s = (w_i * np.log(p_i) for w_i, p_i in zip(weights, clipped_precision_score))\n",
    "    s = np.exp(np.sum(s))\n",
    "    return s"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 3: Computing the BLEU score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def bleu_score(reference, candidate):\n",
    "    BP = brevity_penalty(reference, candidate)\n",
    "    precision = clipped_precision(reference, candidate)\n",
    "    return BP * precision"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 4: Testing with our Example Reference and Candidates Sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results reference versus candidate 1 our own code BLEU:  27.4\n",
      "Results reference versus candidate 2 our own code BLEU:  35.0\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    \"Results reference versus candidate 1 our own code BLEU: \",\n",
    "    round(bleu_score(tokenized_ref, tokenized_cand_1) * 100, 1),\n",
    ")\n",
    "print(\n",
    "    \"Results reference versus candidate 2 our own code BLEU: \",\n",
    "    round(bleu_score(tokenized_ref, tokenized_cand_2) * 100, 1),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### STEP 5: Comparing the Results from our Code with the SacreBLEU Library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results reference versus candidate 1 sacrebleu library sentence BLEU:  27.4\n",
      "Results reference versus candidate 2 sacrebleu library sentence BLEU:  35.0\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    \"Results reference versus candidate 1 sacrebleu library sentence BLEU: \",\n",
    "    round(sacrebleu.corpus_bleu(reference, candidate_1).score, 1),\n",
    ")\n",
    "print(\n",
    "    \"Results reference versus candidate 2 sacrebleu library sentence BLEU: \",\n",
    "    round(sacrebleu.corpus_bleu(reference, candidate_2).score, 1),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 2:  BLEU computation on a corpus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Data Sets for Evaluation Using the BLEU Score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will show a simple pipeline for evaluating machine translated text. Due to storage and speed constraints, we will not be using our own model in this lab (you'll get to do that in the assignment!). Instead, we will be using [Google Translate](https://translate.google.com) to generate English to German translations and we will evaluate it against a known evaluation set. There are three files we will need:\n",
    "\n",
    "1. A source text in English. In this lab, we will use the first 1671 words of the [wmt19](http://statmt.org/wmt19/translation-task.html) evaluation dataset downloaded via SacreBLEU. We just grabbed a subset because of limitations in the number of words that can be translated using Google Translate. \n",
    "2. A reference translation to German of the corresponding first 1671 words from the original English text. This is also provided by SacreBLEU.\n",
    "3. A candidate machine translation to German from the same 1671 words. This is generated by feeding the source text to a machine translation model. As mentioned above, we will use Google Translate to generate the translations in this file.\n",
    "\n",
    "With that, we can now compare the reference an candidate translation to get the BLEU Score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Loading the raw data\n",
    "wmt19news_src = open(\"wmt19_src.txt\", \"rU\")\n",
    "wmt19news_src_1 = wmt19news_src.read()\n",
    "wmt19news_src.close()\n",
    "wmt19news_ref = open(\"wmt19_ref.txt\", \"rU\")\n",
    "wmt19news_ref_1 = wmt19news_ref.read()\n",
    "wmt19news_ref.close()\n",
    "wmt19news_can = open(\"wmt19_can.txt\", \"rU\")\n",
    "wmt19news_can_1 = wmt19news_can.read()\n",
    "wmt19news_can.close()\n",
    "\n",
    "# Tokenizing the raw data\n",
    "tokenized_corpus_src = nltk.word_tokenize(wmt19news_src_1.lower())\n",
    "tokenized_corpus_ref = nltk.word_tokenize(wmt19news_ref_1.lower())\n",
    "tokenized_corpus_cand = nltk.word_tokenize(wmt19news_can_1.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspecting the first sentence of the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "English source text:\n",
      "\n",
      "\n",
      "﻿Welsh AMs worried about 'looking like muppets'\n",
      "There is consternation among some AMs at a suggestion their title should change to MWPs (Member of the Welsh Parliament).\n",
      " -> ['\\ufeffwelsh', 'ams', 'worried', 'about', \"'looking\", 'like', \"muppets'\", 'there', 'is', 'consternation', 'among', 'some', 'ams', 'at', 'a', 'suggestion', 'their', 'title', 'should', 'change', 'to', 'mwps', '(', 'member', 'of', 'the', 'welsh', 'parliament', ')', '.']\n",
      "\n",
      "\n",
      "German reference translation:\n",
      "\n",
      "\n",
      "﻿Walisische Ageordnete sorgen sich \"wie Dödel auszusehen\"\n",
      "Es herrscht Bestürzung unter einigen Mitgliedern der Versammlung über einen Vorschlag, der ihren Titel zu MWPs (Mitglied der walisischen Parlament) ändern soll.\n",
      " -> ['\\ufeffwalisische', 'ageordnete', 'sorgen', 'sich', '``', 'wie', 'dödel', 'auszusehen', \"''\", 'es', 'herrscht', 'bestürzung', 'unter', 'einigen', 'mitgliedern', 'der', 'versammlung', 'über', 'einen', 'vorschlag', ',', 'der', 'ihren', 'titel', 'zu', 'mwps', '(', 'mitglied', 'der', 'walisischen', 'parlament', ')', 'ändern', 'soll', '.']\n",
      "\n",
      "\n",
      "German machine translation:\n",
      "\n",
      "\n",
      "﻿Walisische AMs machten sich Sorgen, dass sie wie Muppets aussehen könnten\n",
      "Einige AMs sind bestürzt über den Vorschlag, ihren Titel in MWPs (Mitglied des walisischen Parlaments) zu ändern.\n",
      "Es ist auf -> ['\\ufeffwalisische', 'ams', 'machten', 'sich', 'sorgen', ',', 'dass', 'sie', 'wie', 'muppets', 'aussehen', 'könnten', 'einige', 'ams', 'sind', 'bestürzt', 'über', 'den', 'vorschlag', ',', 'ihren', 'titel', 'in', 'mwps', '(', 'mitglied', 'des', 'walisischen', 'parlaments']\n"
     ]
    }
   ],
   "source": [
    "print(\"English source text:\")\n",
    "print(\"\\n\")\n",
    "print(f\"{wmt19news_src_1[0:170]} -> {tokenized_corpus_src[0:30]}\")\n",
    "print(\"\\n\")\n",
    "print(\"German reference translation:\")\n",
    "print(\"\\n\")\n",
    "print(f\"{wmt19news_ref_1[0:219]} -> {tokenized_corpus_ref[0:35]}\")\n",
    "print(\"\\n\")\n",
    "print(\"German machine translation:\")\n",
    "print(\"\\n\")\n",
    "print(f\"{wmt19news_can_1[0:199]} -> {tokenized_corpus_cand[0:29]}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results reference versus candidate 1 our own BLEU implementation:  23.6\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    \"Results reference versus candidate 1 our own BLEU implementation: \",\n",
    "    round(bleu_score(tokenized_corpus_ref, tokenized_corpus_cand) * 100, 1),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Results reference versus candidate 1 sacrebleu library BLEU:  23.5\n"
     ]
    }
   ],
   "source": [
    "print(\n",
    "    \"Results reference versus candidate 1 sacrebleu library BLEU: \",\n",
    "    round(sacrebleu.corpus_bleu(wmt19news_ref_1, wmt19news_can_1).score, 1),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**BLEU Score Interpretation on a Corpus**\n",
    "\n",
    "|Score      | Interpretation                                                |\n",
    "|:---------:|:-------------------------------------------------------------:|\n",
    "| < 10      | Almost useless                                                |\n",
    "| 10 - 19   | Hard to get the gist                                          |\n",
    "| 20 - 29   | The gist is clear, but has significant grammatical errors     |\n",
    "| 30 - 40   | Understandable to good translations                           |\n",
    "| 40 - 50   | High quality translations                                     |\n",
    "| 50 - 60   | Very high quality, adequate, and fluent translations          |\n",
    "| > 60      | Quality often better than human                               |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the table above (taken [here](https://cloud.google.com/translate/automl/docs/evaluate)), we can see the gist of the translation is clear but has significant grammatical errors. Nonetheless, the results of our coded BLEU score are almost identical to those of the SacreBLEU package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "encoding": "# -*- coding: utf-8 -*-",
   "formats": "ipynb,py:percent"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}