pytorch_enhance2_embedding_explained.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "embedding_explained.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/raynardj/python4ml/blob/master/pytorch_enhance2_embedding_explained.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GPvD5LItosra",
        "colab_type": "text"
      },
      "source": [
        "# Embedding Syntex in Numpy and Pytorch"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9BNXCSuilLFE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from torch import nn\n",
        "import torch\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mUp13mDemGma",
        "colab_type": "text"
      },
      "source": [
        "### Numpy First"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OlAjbMmDmJLl",
        "colab_type": "text"
      },
      "source": [
        "* Discrete data, it usually means index and categorical information only.\n",
        "\n",
        "* Most frameworks can fit any thing, but only contineous data.\n",
        "\n",
        "* For discrete data, embedding is a bridge that we can quantify categorical data into contineous ones"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lP3acK4vlfC-",
        "colab_type": "code",
        "outputId": "775d2a12-4530-4c97-d6bc-7646ca826e94",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "data = np.random.choice(range(7),100)\n",
        "data"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([4, 5, 1, 4, 3, 1, 6, 5, 0, 5, 5, 4, 2, 1, 4, 6, 4, 2, 0, 6, 0, 5,\n",
              "       2, 1, 3, 5, 5, 5, 5, 4, 1, 6, 4, 4, 5, 6, 4, 4, 2, 4, 5, 2, 2, 2,\n",
              "       5, 3, 6, 4, 5, 5, 1, 1, 0, 1, 2, 4, 3, 5, 2, 2, 0, 4, 6, 2, 0, 3,\n",
              "       2, 1, 3, 4, 1, 2, 2, 1, 3, 6, 6, 0, 6, 5, 0, 0, 1, 6, 4, 3, 4, 2,\n",
              "       0, 5, 2, 5, 2, 5, 0, 3, 6, 3, 1, 6])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V6nk-pkWmq2E",
        "colab_type": "text"
      },
      "source": [
        "We set latent factor to 5, initialize them with random number.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5AXSVsUjmiyR",
        "colab_type": "code",
        "outputId": "dc154792-4dfb-4594-ad87-859a5fab5f88",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "embedding = np.random.rand(7,5)\n",
        "embedding"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[0.89699559, 0.54985025, 0.00200458, 0.16661117, 0.14311747],\n",
              "       [0.57630332, 0.87428202, 0.5086325 , 0.57562626, 0.45423915],\n",
              "       [0.17084596, 0.50939843, 0.54167113, 0.67763697, 0.78783203],\n",
              "       [0.01597522, 0.09425785, 0.92997523, 0.02148565, 0.46372017],\n",
              "       [0.40199222, 0.15242902, 0.86390534, 0.95004193, 0.03541003],\n",
              "       [0.40891402, 0.92982274, 0.54180708, 0.52751166, 0.0679205 ],\n",
              "       [0.75862153, 0.92568992, 0.51646221, 0.73440811, 0.63432683]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rUV2HUnBvTz4",
        "colab_type": "text"
      },
      "source": [
        "We can find the latent vector by indexing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-a7P8Mp9mxBi",
        "colab_type": "code",
        "outputId": "9ab4d884-5249-4057-eaee-a4e2ae4c872f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "embedding[5]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.40891402, 0.92982274, 0.54180708, 0.52751166, 0.0679205 ])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bM2IiwJ_mxFB",
        "colab_type": "code",
        "outputId": "5cd9854a-0cbe-41b2-b8e7-db0dcdc75c5a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "embedding[6]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([0.75862153, 0.92568992, 0.51646221, 0.73440811, 0.63432683])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XnN1vBnOmxH1",
        "colab_type": "code",
        "outputId": "9b02dcb5-1633-4133-ff27-a5f37c5902f3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "embedding[data].shape"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(100, 5)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FDbneViXnALQ",
        "colab_type": "text"
      },
      "source": [
        "### In PyTorch\n",
        "The initialized random number will \n",
        "* have easier time get updated by optimizer\n",
        "* gradient calculation could be automated\n",
        "* supports gpu"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wi0x0Ylcl3oK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data_x = torch.LongTensor(data)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5sCBTuNGnVC3",
        "colab_type": "code",
        "outputId": "d89856b8-7e4a-4dff-bb1c-b13044269bd3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "data_x"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([4, 5, 1, 4, 3, 1, 6, 5, 0, 5, 5, 4, 2, 1, 4, 6, 4, 2, 0, 6, 0, 5, 2, 1,\n",
              "        3, 5, 5, 5, 5, 4, 1, 6, 4, 4, 5, 6, 4, 4, 2, 4, 5, 2, 2, 2, 5, 3, 6, 4,\n",
              "        5, 5, 1, 1, 0, 1, 2, 4, 3, 5, 2, 2, 0, 4, 6, 2, 0, 3, 2, 1, 3, 4, 1, 2,\n",
              "        2, 1, 3, 6, 6, 0, 6, 5, 0, 0, 1, 6, 4, 3, 4, 2, 0, 5, 2, 5, 2, 5, 0, 3,\n",
              "        6, 3, 1, 6])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pcK_-hEynWMT",
        "colab_type": "code",
        "outputId": "a96293a2-ffa1-4482-ef16-e03ea9f73f7d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "source": [
        "embedding_ = nn.Embedding(7,5)\n",
        "list(embedding_.parameters())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Parameter containing:\n",
              " tensor([[-2.0520,  0.2496, -1.2246,  2.2228, -0.4564],\n",
              "         [-0.1845, -0.6931, -0.4337, -1.4748,  0.1549],\n",
              "         [ 0.2013,  1.3846, -1.9696, -0.2395, -1.7107],\n",
              "         [ 0.3187,  1.5072, -0.4693,  0.6171, -1.4572],\n",
              "         [ 0.4569,  0.4616, -1.6014,  0.2157,  0.0691],\n",
              "         [-0.2155,  0.5961,  1.8220,  0.3629,  0.3233],\n",
              "         [-0.9444,  0.4652, -0.3823,  1.4068, -0.5690]], requires_grad=True)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 22
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GNRL7RWanfiV",
        "colab_type": "code",
        "outputId": "ebe114a5-4fe3-4561-e423-3701e1366b46",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "x1 = embedding_(torch.LongTensor([0]))\n",
        "x1"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-2.0520,  0.2496, -1.2246,  2.2228, -0.4564]],\n",
              "       grad_fn=<EmbeddingBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wtRIztJ7nz46",
        "colab_type": "code",
        "outputId": "6062d336-9f39-4e4a-afe7-4d462db829b2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "x2 = embedding_(torch.LongTensor([1]))\n",
        "x2"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-0.1845, -0.6931, -0.4337, -1.4748,  0.1549]],\n",
              "       grad_fn=<EmbeddingBackward>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zQ1YCV9zn-xr",
        "colab_type": "code",
        "outputId": "d62cdd72-dc0b-4b11-b229-0bc8e8facff6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "test_loss = torch.pow(x1-x2,2).mean()\n",
        "test_loss"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor(3.8097, grad_fn=<MeanBackward0>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S-4-ZTb4oU-n",
        "colab_type": "text"
      },
      "source": [
        "Auto calculate gradient"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CfXZ79WboSSw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "test_loss.backward()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yRC6zCl8oYAE",
        "colab_type": "code",
        "outputId": "b1064286-8014-4bed-95da-16c5b29b29fe",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "source": [
        "list(embedding_.parameters())[0].grad"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[-0.7470,  0.3771, -0.3164,  1.4791, -0.2445],\n",
              "        [ 0.7470, -0.3771,  0.3164, -1.4791,  0.2445],\n",
              "        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],\n",
              "        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],\n",
              "        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],\n",
              "        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000],\n",
              "        [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bRRH4WkN1jG8",
        "colab_type": "code",
        "outputId": "8ea5109b-03b1-4edc-ab00-000834e319ea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "embedding_(data_x).size()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([100, 5])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 36
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YiWXCguKob6B",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "x_conti = torch.rand(100,3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ipw04uPB1bNQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# concatenate on last dimension\n",
        "x_all = torch.cat([x_conti,embedding_(data_x)] ,dim=-1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "A5zPwRN81xn-",
        "colab_type": "code",
        "outputId": "2a543f6e-1442-4c86-d560-e802a11193ef",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "x_all.size()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([100, 8])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 40
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T60XTAlR14eF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}