The_ultimate_guide_to_Encoder_Decoder_Models_4_4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "The ultimate guide to Encoder Decoder Models 4/4",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nsankar/AutoTS/blob/master/The_ultimate_guide_to_Encoder_Decoder_Models_4_4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DRzbFeCCZBw1"
      },
      "source": [
        "%%capture\n",
        "!pip install -qq git+https://github.com/huggingface/transformers.git"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "myjTH2wFeWTU"
      },
      "source": [
        "# **The ultimate guide to Encoder Decoder Models 4/4**\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U9YCIjCPInpe"
      },
      "source": [
        "The *transformer-based* encoder-decoder model was introduced by Vaswani et al. in the famous [Attention is all you need paper](https://arxiv.org/abs/1706.03762) and is today the *de-facto* standard encoder-decoder architecture in natural language processing (NLP).\n",
        "\n",
        "Recently, there has been a lot of research on different *pre-training* objectives for transformer-based encoder-decoder models, *e.g.* T5, Bart, Pegasus, ProphetNet, Marge, *etc*..., but the model architecture has stayed largely the same.\n",
        "\n",
        "The goal of the blog post is to give an **in-detail** explanation of **how** the transformer-based encoder-decoder architecture models *sequence-to-sequence* problems. We will focus on the mathematical model defined by the architecture and how the model can be used in inference. Along the way, we will give some background on sequence-to-sequence models in NLP and break down the *transformer-based* encoder-decoder architecture into its **encoder** and **decoder** part. We provide many illustrations and establish the link\n",
        "between the theory of *transformer-based* encoder-decoder models and their practical usage in 🤗Transformers for inference.\n",
        "Note that this blog post does *not* explain how such models can be trained - this will be the topic of a future blog post.\n",
        "\n",
        "Transformer-based encoder-decoder models are the result of years of research on *representation learning* and *model architectures*. \n",
        "This notebook provides a short summary of the history of neural encoder-decoder models. For more context, the reader is advised to read this awesome [blog post](https://ruder.io/a-review-of-the-recent-history-of-nlp/) by Sebastion Ruder. Additionally, a basic understanding of the *self-attention architecture* is recommended. \n",
        "The following blog post by Jay Alammar serves as a good refresher on the original Transformer model [here](http://jalammar.github.io/illustrated-transformer/).\n",
        "\n",
        "At the time of writing this notebook, 🤗Transformers comprises the encoder-decoder models *T5*, *Bart*, *MarianMT*, and *Pegasus*, which are summarized in the docs under [model summaries](https://huggingface.co/transformers/model_summary.html#sequence-to-sequence-models).\n",
        "\n",
        "The notebook is divided into four parts:\n",
        "\n",
        "- **Background** - *A short history of neural encoder-decoder models is given with a focus on on RNN-based models.* - [click here](https://colab.research.google.com/drive/18ZBlS4tSqSeTzZAVFxfpNDb_SrZfAOMf?usp=sharing)\n",
        "- **Encoder-Decoder** - *The transformer-based encoder-decoder model is presented and it is explained how the model is used for inference.* - [click here](https://colab.research.google.com/drive/1XpKHijllH11nAEdPcQvkpYHCVnQikm9G?usp=sharing)\n",
        "- **Encoder** - *The encoder part of the model is explained in detail.* - [click here](https://colab.research.google.com/drive/1HJhnWMFizEKKWEAb-k7QDBv4c03hXbCR?usp=sharing)\n",
        "- **Decoder** - *The decoder part of the model is explained in detail.*\n",
        "\n",
        "Each part builds upon the previous part, but can also be read on its own. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FYItWQClOCX2"
      },
      "source": [
        "## **Decoder**\n",
        "\n",
        "As mentioned in the *Encoder-Decoder* section, the *transformer-based* decoder defines the conditional probability distribution of a target sequence given the contextualized encoding sequence:\n",
        "\n",
        "$$ p_{\\theta_{dec}}(\\mathbf{Y}_{1: m} | \\mathbf{\\overline{X}}_{1:n}), $$\n",
        "\n",
        "which by Bayes' rule can be decomposed into a product of conditional distributions of the next target vector given the contextualized encoding sequence and all previous target vectors:\n",
        "\n",
        "$$ p_{\\theta_{dec}}(\\mathbf{Y}_{1:m} | \\mathbf{\\overline{X}}_{1:n}) = \\prod_{i=1}^{m} p_{\\theta_{dec}}(\\mathbf{y}_i | \\mathbf{Y}_{0: i-1}, \\mathbf{\\overline{X}}_{1:n}). $$\n",
        "\n",
        "Let's first understand how the transformer-based decoder defines a probability distribution. The transformer-based decoder is a stack of *decoder blocks* followed by a dense layer, the \"LM head\".\n",
        "The stack of decoder blocks maps the contextualized encoding sequence $\\mathbf{\\overline{X}}_{1:n}$ and a target vector sequence prepended by the $\\text{BOS}$ vector and cut to the last target vector, *i.e.* $\\mathbf{Y}_{0:i-1}$, to an encoded sequence of target vectors $\\mathbf{\\overline{Y}}_{0: i-1}$. Then, the \"LM head\" maps the encoded sequence of target vectors $\\mathbf{\\overline{Y}}_{0: i-1}$ to a sequence of logit vectors $\\mathbf{L}_{1:n} = \\mathbf{l}_1, \\ldots, \\mathbf{l}_n$, whereas the dimensionality of each logit vector $\\mathbf{l}_i$ corresponds to the size of the vocabulary. This way, for each $i \\in \\{1, \\ldots, n\\}$ a probability distribution over the whole vocabulary can be obtained by applying a softmax operation on $\\mathbf{l}_i$. These distributions define the conditional distribution: \n",
        "\n",
        "$$p_{\\theta_{dec}}(\\mathbf{y}_i | \\mathbf{Y}_{0: i-1}, \\mathbf{\\overline{X}}_{1:n}), \\forall i \\in \\{1, \\ldots, n\\},$$ \n",
        "\n",
        "respectively. The \"LM head\" is often tied to the transpose of the word embedding matrix, *i.e.* $\\mathbf{W}_{\\text{emb}}^{\\intercal} = \\left[\\mathbf{y}^1, \\ldots, \\mathbf{y}^{\\text{vocab}}\\right]^{\\intercal}$ ${}^1$. Intuitively this means that for all $i \\in \\{0, \\ldots, n - 1\\}$ the \"LM Head\" layer compares the encoded output vector $\\mathbf{\\overline{y}}_i$ to all word embeddings in the vocabulary $\\mathbf{y}^1, \\ldots, \\mathbf{y}^{\\text{vocab}}$ so that the logit vector $\\mathbf{l}_{i+1}$ represents the similarity scores between the encoded output vector and each word embedding. The softmax operation simply transformers the similarity scores to a probability distribution. For each $i \\in \\{1, \\ldots, n\\}$, the following equations hold:\n",
        "\n",
        "$$ p_{\\theta_{dec}}(\\mathbf{y} | \\mathbf{\\overline{X}}_{1:n}, \\mathbf{Y}_{0:i-1})$$\n",
        "$$ = \\text{Softmax}(f_{\\theta_{\\text{dec}}}(\\mathbf{\\overline{X}}_{1:n}, \\mathbf{Y}_{0:i-1}))$$\n",
        "$$ = \\text{Softmax}(\\mathbf{W}_{\\text{emb}}^{\\intercal} \\mathbf{\\overline{y}}_{i-1})$$\n",
        "$$ = \\text{Softmax}(\\mathbf{l}_i). $$\n",
        "\n",
        "Putting it all together, in order to model the conditional distribution of a target vector sequence $\\mathbf{Y}_{1: m}$, the target vectors $\\mathbf{Y}_{1:m-1}$ prepended by the special $\\text{BOS}$ vector, *i.e.* $\\mathbf{y}_0$, are first mapped together with the contextualized encoding sequence $\\mathbf{\\overline{X}}_{1:n}$ to the logit vector sequence $\\mathbf{L}_{1:m}$. \n",
        "Consequently, each logit target vector $\\mathbf{l}_i$ is transformed into a conditional probability distribution of the target vector $\\mathbf{y}_i$ using the softmax operation. Finally, the conditional probabilities of all target vectors $\\mathbf{y}_1, \\ldots, \\mathbf{y}_m$ multiplied together to yield the conditional probability of the complete target vector sequence:\n",
        "\n",
        "$$ p_{\\theta_{dec}}(\\mathbf{Y}_{1:m} | \\mathbf{\\overline{X}}_{1:n}) = \\prod_{i=1}^{m} p_{\\theta_{dec}}(\\mathbf{y}_i | \\mathbf{Y}_{0: i-1}, \\mathbf{\\overline{X}}_{1:n}).$$\n",
        "\n",
        "In contrast to transformer-based encoders, in transformer-based decoders, the encoded output vector $\\mathbf{\\overline{y}}_i$ should be a good representation of the *next* target vector $\\mathbf{y}_{i+1}$ and not of the input vector itself. Additionally, the encoded output vector $\\mathbf{\\overline{y}}_i$ should be conditioned on all contextualized encoding sequence $\\mathbf{\\overline{X}}_{1:n}$. To meet these requirements each decoder block consists of a **uni-directional** self-attention layer, followed by a **cross-attention** layer and two feed-forward layers ${}^2$.\n",
        "The uni-directional self-attention layer puts each of its input vectors $\\mathbf{y'}_j$ only into relation with all previous input vectors $\\mathbf{y'}_i, \\text{ with } i \\le q$ for all $j \\in \\{1, \\ldots, n\\}$ to model the probability distribution of the next target vectors. \n",
        "The cross-attention layer puts each of its input vectors $\\mathbf{y''}_j$ into relation with all contextualized encoding vectors $\\mathbf{\\overline{X}}_{1:n}$ to condition the probability distribution of the next target vectors on the input of the encoder as well.\n",
        "\n",
        "Alright, let's visualize the *transformer-based* decoder for our English to German translation example.\n",
        "\n",
        "![](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/encoder_decoder/encoder_decoder_detail.png)\n",
        "\n",
        "We can see that the decoder maps the input $\\mathbf{Y}_{0:5}$ \"BOS\", \"Ich\", \"will\", \"ein\", \"Auto\", \"kaufen\" (shown in light red) together with the contextualized sequence of \"I\", \"want\", \"to\", \"buy\", \"a\", \"car\", \"EOS\", *i.e.* $\\mathbf{\\overline{X}}_{1:7}$ (shown in dark green) to the logit vectors $\\mathbf{L}_{1:6}$ (shown in dark red).\n",
        "\n",
        "Applying a softmax operation on each $\\mathbf{l}_1, \\mathbf{l}_2, \\ldots, \\mathbf{l}_5$ can thus define the conditional probability distributions:\n",
        "\n",
        "$$ p_{\\theta_{dec}}(\\mathbf{y} | \\text{BOS}, \\mathbf{\\overline{X}}_{1:7}), $$\n",
        "$$ p_{\\theta_{dec}}(\\mathbf{y} | \\text{BOS Ich}, \\mathbf{\\overline{X}}_{1:7}), $$\n",
        "$$ \\ldots, $$\n",
        "$$ p_{\\theta_{dec}}(\\mathbf{y} | \\text{BOS Ich will ein Auto kaufen}, \\mathbf{\\overline{X}}_{1:7}). $$\n",
        "\n",
        "The overall conditional probability of:\n",
        "\n",
        "$$ p_{\\theta_{dec}}(\\text{Ich will ein Auto kaufen EOS} | \\mathbf{\\overline{X}}_{1:n})$$\n",
        "\n",
        "can therefore be computed as the following product:\n",
        "\n",
        "$$ p_{\\theta_{dec}}(\\text{Ich} | \\text{BOS}, \\mathbf{\\overline{X}}_{1:7}) \\times \\ldots \\times p_{\\theta_{dec}}(\\text{EOS} | \\text{BOS Ich will ein Auto kaufen}, \\mathbf{\\overline{X}}_{1:7}). $$\n",
        "\n",
        "The red box on the right shows a decoder block for the first three target vectors $\\mathbf{y}_0, \\mathbf{y}_1, \\mathbf{y}_2$. In the lower part, the uni-directional self-attention mechanism is illustrated and in the middle, the cross-attention mechanism is illustrated. Let's first focus on uni-directional self-attention.\n",
        "\n",
        "As in bi-directional self-attention, in uni-directional self-attention, the query vectors $\\mathbf{q}_0, \\ldots, \\mathbf{q}_{m-1}$ (shown in purple below), key vectors $\\mathbf{k}_0, \\ldots, \\mathbf{k}_{m-1}$ (shown in orange below), and value vectors $\\mathbf{v}_0, \\ldots, \\mathbf{v}_{m-1}$ (shown in blue below) are projected from their respective input vectors $\\mathbf{y'}_0, \\ldots, \\mathbf{y}_{m-1}$ (shown in light red below). However, in uni-directional self-attention, each query vector $\\mathbf{q}_i$ is compared *only* to its respective key vector and all previous ones, namely $\\mathbf{k}_0, \\ldots, \\mathbf{k}_i$ to yield the respective *attention weights*.\n",
        "This prevents an output vector $\\mathbf{y''}_j$ (shown in dark red below) to include any information about the following input vector $\\mathbf{y}_i, \\text{ with } i > 1$ for all $j \\in \\{0, \\ldots, m - 1 \\}$. As is the case in bi-directional self-attention, the attention weights are then multiplied by their respective value vectors and summed together.\n",
        "\n",
        "We can summarize uni-directional self-attention as follows: \n",
        "\n",
        "$$\\mathbf{y''}_i = \\mathbf{V}_{0: i} \\textbf{Softmax}(\\mathbf{K}_{0: i}^\\intercal \\mathbf{q}_i) + \\mathbf{y'}_i. $$\n",
        "\n",
        "Note that the index range of the key and value vectors is $0:i$ instead of $0: m-1$ which would be the range of the key vectors in bi-directional self-attention.\n",
        "\n",
        "Let's illustrate uni-directional self-attention for the input vector $\\mathbf{y'}_1$ for our example above.\n",
        "\n",
        "![](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/encoder_decoder/causal_attn.png)\n",
        "\n",
        "As can be seen $\\mathbf{y''}_1$ only depends on $\\mathbf{y'}_0$ and $\\mathbf{y'}_1$. Therefore, we put the vector representation of the word \"Ich\", *i.e.* $\\mathbf{y'}_1$ only into relation with itself and the \"BOS\" target vector, *i.e.* $\\mathbf{y'}_0$, but **not** with the vector representation of the word \"will\", *i.e.* $\\mathbf{y'}_2$.\n",
        "\n",
        "So why is it important that we use uni-directional self-attention in the decoder instead of bi-directional self-attention? \n",
        "As stated above, a transformer-based decoder defines a mapping from a sequence of input vector $\\mathbf{Y}_{0: m-1}$ to the logits corresponding to the **next** decoder input vectors, namely $\\mathbf{L}_{1:m}$.\n",
        "In our example, this means, *e.g.* that the input vector $\\mathbf{y}_1$ = \"Ich\" is mapped to the logit vector $\\mathbf{l}_2$, which is then used to predict the input vector $\\mathbf{y}_2$. Thus, if $\\mathbf{y'}_1$ would have access to the following input vectors $\\mathbf{Y'}_{2:5}$, the decoder would simply copy the vector representation of \"will\", *i.e.* $\\mathbf{y'}_2$, to be its output $\\mathbf{y''}_1$. This would be forwarded to the last layer so that the encoded output vector $\\mathbf{\\overline{y}}_1$ would essentially just correspond to the vector representation $\\mathbf{y}_2$. \n",
        "\n",
        "This is obviously disadvantageous as the transformer-based decoder would never learn to predict the next word given all previous words, but just copy the target vector $\\mathbf{y}_i$ through the network to $\\mathbf{\\overline{y}}_{i-1}$ for all $i \\in \\{1, \\ldots, m \\}$. In order to define a conditional distribution of the next target vector, the distribution cannot be conditioned on the next target vector itself. It does not make much sense to predict $\\mathbf{y}_i$ from $p(\\mathbf{y} | \\mathbf{Y}_{0:i}, \\mathbf{\\overline{X}})$ because the distribution is conditioned on the target vector it is supposed to model.\n",
        "The uni-directional self-attention architecture, therefore, allows us to define a *causal* probability distribution, which is necessary to effectively model a conditional distribution of the next target vector.\n",
        "\n",
        "Great! Now we can move to the layer that connects the encoder and decoder - the *cross-attention* mechanism!\n",
        "\n",
        "The cross-attention layer takes two vector sequences as inputs: the outputs of the uni-directional self-attention layer, *i.e.* $\\mathbf{Y''}_{0: m-1}$ and the contextualized encoding vectors $\\mathbf{\\overline{X}}_{1:n}$.\n",
        "As in the self-attention layer, the query vectors $\\mathbf{q}_0, \\ldots, \\mathbf{q}_{m-1}$ are projections of the output vectors of the previous layer, *i.e.* $\\mathbf{Y''}_{0: m-1}$. However, the key and value vectors $\\mathbf{k}_0, \\ldots, \\mathbf{k}_{m-1}$ and $\\mathbf{v}_0, \\ldots, \\mathbf{v}_{m-1}$ are projections of the contextualized encoding vectors $\\mathbf{\\overline{X}}_{1:n}$. Having defined key, value, and query vectors, a query vector $\\mathbf{q}_i$ is then compared to *all* key vectors and the corresponding score is used to weight the respective value vectors, just as is the case for *bi-directional* self-attention to give the output vector $\\mathbf{y'''}_i$ for all $i \\in \\{0, \\ldots, m-1\\}. Cross-attention can be summarized as follows:\n",
        "\n",
        "$$\n",
        "\\mathbf{y'''}_i = \\mathbf{V}_{1:n} \\textbf{Softmax}(\\mathbf{K}_{1: n}^\\intercal \\mathbf{q}_i) + \\mathbf{y''}_i.\n",
        "$$\n",
        "\n",
        "Note that the index range of the key and value vectors is $1:n$ corresponding to the number of contextualized encoding vectors.\n",
        "\n",
        "Let's visualize the cross-attention mechanism Let's for the input vector $\\mathbf{y''}_1$ for our example above.\n",
        "\n",
        "![](https://raw.githubusercontent.com/patrickvonplaten/scientific_images/master/encoder_decoder/cross_attention.png)\n",
        "\n",
        "We can see that the query vector $\\mathbf{q}_1$ (shown in purple) is derived from $\\mathbf{y''}_1$ and therefore relies on a vector representation of the word \"Ich\". The query vector $\\mathbf{q}_1$ (shown in red) is then compared to the key vectors $\\mathbf{k}_1, \\ldots, \\mathbf{k}_7$ (shown in yellow) corresponding to the contextual encoding representation of all encoder input vectors $\\mathbf{X}_{1:n}$ = \"I want to buy a car EOS\". This puts the vector representation of \"Ich\" into direct relation with all encoder input vectors. Finally, the attention weights are multiplied by the value vectors $\\mathbf{v}_1, \\ldots, \\mathbf{v}_7$ (shown in turquoise) to yield in addition to the input vector $\\mathbf{y''}_1$ the output vector $\\mathbf{y'''}_1$ (shown in dark red). \n",
        "\n",
        "So intuitively, what happens here exactly? \n",
        "Each output vector $\\mathbf{y'}_i$ is a weighted sum of all value projections of the encoder inputs $\\mathbf{v}_{1}, \\ldots, \\mathbf{v}_7$ plus the input vector itself $\\mathbf{y}_i$ (*c.f.* illustrated formula above). The key mechanism to understand is the following: Depending on how similar a query projection of the *input decoder vector* $\\mathbf{q}_i$ is to a key projection of the *encoder input vector* $\\mathbf{k}_j$, the more important is the value projection of the encoder input vector $\\mathbf{v}_j$. In loose terms this means, the more \"related\" a decoder input representation is to an encoder input representation, the more does the input representation influence the decoder output representation. \n",
        "\n",
        "Cool! Now we can see how this architecture nicely conditions each output vector $\\mathbf{y'''}_i$ on the interaction between the encoder input vectors $\\mathbf{\\overline{X}}_{1:n}$ and the input vector $\\mathbf{y''}_i$. \n",
        "Another important observation at this point is that the architecture is completely independent of the number $n$ of contextualized encoding vectors $\\mathbf{\\overline{X}}_{1:n}$ on which the output vector $\\mathbf{y'''}_i$ is conditioned on. All projection matrices $\\mathbf{W}^{\\text{cross}}_{k}$ and $\\mathbf{W}^{\\text{cross}}_{v}$ to derive the key vectors $\\mathbf{k}_1, \\ldots, \\mathbf{k}_n$ and the value vectors $\\mathbf{v}_1, \\ldots, \\mathbf{v}_n$ respectively are shared across all positions $1, \\ldots, n$ and all value vectors $\\mathbf{v}_1, \\ldots, \\mathbf{v}_n$ are summed together to a single weighted averaged vector.\n",
        "Now it becomes obvious as well, why the transformer-based decoder does not suffer from the long-range dependency problem, the RNN-based decoder suffers from. Because each decoder logit vector is *directly* dependent on every single encoded output vector, the number of mathematical operations to compare the first encoded output vector and the last decoder logit vector amounts essentially to just one.\n",
        "\n",
        "To conclude, the uni-directional self-attention layer is responsible for conditioning each output vector on all previous decoder input vectors and the current input vector and the cross-attention layer is responsible to further condition each output vector on all encoded input vectors.\n",
        "\n",
        "To verify our theoretical understanding, let's continue our code example from the encoder section above.\n",
        "\n",
        "---\n",
        "${}^1$ The word embedding matrix $\\mathbf{W}_{\\text{emb}}$ gives each input word a unique *context-independent* vector representation. This matrix is often fixed as the \"LM Head\" layer. However, the \"LM Head\" layer can very well consist of a completely independent \"encoded vector-to-logit\" weight mapping.\n",
        "\n",
        "${}^2$ Again, an in-detail explanation of the role the feed-forward layers play in transformer-based models is out-of-scope for this notebook. It is argued in [Yun et. al, (2017)](https://arxiv.org/pdf/1912.10077.pdf) that feed-forward layers are crucial to map each contextual vector $\\mathbf{x'}_i$ individually to the desired output space, which the *self-attention* layer does not manage to do on its own. It should be noted here, that each output token $\\mathbf{x'}$ is processed by the same feed-forward layer. For more detail, the reader is advised to read the paper.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8iZC1-5bb_hQ",
        "outputId": "3b8b6393-b7f0-40e9-8f7d-1d8b7693bfde",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "source": [
        "from transformers import MarianMTModel, MarianTokenizer\n",
        "import torch\n",
        "\n",
        "tokenizer = MarianTokenizer.from_pretrained(\"Helsinki-NLP/opus-mt-en-de\")\n",
        "model = MarianMTModel.from_pretrained(\"Helsinki-NLP/opus-mt-en-de\")\n",
        "embeddings = model.get_input_embeddings()\n",
        "\n",
        "# get encoded input vectors\n",
        "input_ids = tokenizer(\"I want to buy a car\", return_tensors=\"pt\").input_ids\n",
        "encoded_output_vectors = model.base_model.encoder(input_ids, return_dict=True).last_hidden_state\n",
        "\n",
        "# create ids of encoded input vectors\n",
        "decoder_input_ids = tokenizer(\"<pad> Ich will ein\", return_tensors=\"pt\", add_special_tokens=False).input_ids\n",
        "\n",
        "# pass decoder input_ids and encoded input vectors to decoder\n",
        "decoder_output_vectors = model.base_model.decoder(decoder_input_ids, encoded_output_vectors, None, None, None, return_dict=True).last_hidden_state\n",
        "\n",
        "# derive embeddings by multiplying decoder outputs with embedding weights\n",
        "lm_logits = torch.nn.functional.linear(decoder_output_vectors, embeddings.weight, bias=model.final_logits_bias)\n",
        "\n",
        "# change the decoder input slightly\n",
        "decoder_input_ids_perturbed = tokenizer(\"</s> Ich will das\", return_tensors=\"pt\").input_ids\n",
        "decoder_output_vectors_perturbed = model.base_model.decoder(decoder_input_ids, encoded_output_vectors, None, None, None, return_dict=True).last_hidden_state\n",
        "lm_logits_perturbed = torch.nn.functional.linear(decoder_output_vectors_perturbed, embeddings.weight, bias=model.final_logits_bias)\n",
        "\n",
        "# compare shape and encoding of first vector\n",
        "print(f\"Shape of decoder input vectors {embeddings(decoder_input_ids).shape}. Shape of decoder logits {lm_logits.shape}\")\n",
        "\n",
        "# compare values of word embedding of \"I\" for input_ids and perturbed input_ids\n",
        "print(\"Is encoding for `Ich` equal to its perturbed version?: \", torch.allclose(lm_logits[0, 0], lm_logits_perturbed[0, 0], atol=1e-3))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Shape of decoder input vectors torch.Size([1, 5, 512]). Shape of decoder logits torch.Size([1, 5, 58101])\n",
            "Is encoding for `Ich` equal to its perturbed version?:  True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Fb5k_NzGgr8y"
      },
      "source": [
        "We compare the output shape of the decoder input word embeddings, *i.e.* `embeddings(decoder_input_ids)` (corresponds to $\\mathbf{Y}_{0: 4}$, here `<pad>` corresponds to BOS and \"Ich will das\" is tokenized to 4 tokens) with the dimensionality of the `lm_logits`(corresponds to $\\mathbf{L}_{1:5}$).\n",
        "Also, we have passed the word sequence \"<pad> Ich will das\" and a slightly perturbated version \"<pad> Ich will das\" together with the `encoder_output_vectors` through the encoder to check if the second `lm_logit`, corresponding to \"Ich\", differs when only the last word is changed in the input sequence (\"ein\" -> \"das\").\n",
        "\n",
        "As expected the output shapes of the decoder input word embeddings and lm_logits, *i.e.* the dimensionality of $\\mathbf{Y}_{0: 4}$ and $\\mathbf{L}_{1:5}$ are different in the last dimension. While the sequence length is the same (=5), the dimensionality of a decoder input word embedding corresponds to `model.config.hidden_size`, whereas the dimensionality of a `lm_logit` corresponds to the vocabulary size `model.config.vocab_size`, as explained above.\n",
        "Second, it can be noted that the values of the encoded output vector of $\\mathbf{l}_1 = \\text{\"Ich\"}$ are the same when the last word is changed from \"ein\" to \"das\". This however should not come as a surprise if one has understood uni-directional self-attention.\n",
        "\n",
        "On a final side-note, *auto-regressive* models, such as GPT2, have the same architecture as *transformer-based* decoder models **if** one removes the cross-attention layer because stand-alone auto-regressive models are not conditioned on any encoder outputs. \n",
        "So auto-regressive models are essentially the same as *auto-encoding* models but replace bi-directional attention with uni-directional attention. These models can also be pre-trained on massive open-domain text data to show impressive performances on natural language generation (NLG) tasks. In [Radford et al. (2019)](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), the authors show that a pre-trained GPT2 model can achieve SOTA or close to SOTA results on a variety of NLG tasks without much fine-tuning. All *auto-regressive* models of 🤗Transformers can be found [here](https://huggingface.co/transformers/model_summary.html#autoregressive-models).\n",
        "\n",
        "Alright, that's it! Now, you should have gotten a good understanding of *transformer-based* encoder-decoder models and how to use them with the 🤗Transformers library.\n",
        "\n",
        "Thanks a lot to Victor Sanh, Sasha Rush, Sam Shleifer, Oliver Åstrand, \n",
        "‪Ted Moskovitz and Kristian Kyvik for giving valuable feedback."
      ]
    }
  ]
}