Lab06_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6m-8_XuzwSrr"
   },
   "source": [
    "# Lab 6-2: Fancy Softmax Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aeQ2U1QowSrz"
   },
   "source": [
    "Author: Seungjae Lee (이승재)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "15YPxCNCwSr0"
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5hiuKa9zwSr1"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WEe8k4X3wSr3",
    "outputId": "5d602cb4-eb3a-4c57-f774-12a554899749"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<torch._C.Generator at 0x7fb0672ec190>"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# For reproducibility\n",
    "torch.manual_seed(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "phxuZXOpwSr4"
   },
   "source": [
    "## Cross-entropy Loss with `torch.nn.functional`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9DsbHpc6wSr4"
   },
   "source": [
    "PyTorch has `F.log_softmax()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Nm_i-fcJwSr5",
    "outputId": "4ef0cb0f-3249-4987-cddf-d541fe4325ac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[1., 0., 0., 0., 0.],\n",
       "        [0., 0., 1., 0., 0.],\n",
       "        [0., 1., 0., 0., 0.]])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = torch.rand(3, 5, requires_grad=True)\n",
    "hypothesis = F.softmax(z, dim=1)\n",
    "y = torch.randint(5, (3,)).long()\n",
    "y_one_hot = torch.zeros_like(hypothesis)\n",
    "y_one_hot.scatter_(1, y.unsqueeze(1), 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Hy22tAQswSr6",
    "outputId": "4046c8ca-bee9-448e-fd7a-5b50b154b932"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.3301, -1.8084, -1.6846, -1.3530, -2.0584],\n",
       "        [-1.4147, -1.8174, -1.4602, -1.6450, -1.7758],\n",
       "        [-1.5025, -1.6165, -1.4586, -1.8360, -1.6776]], grad_fn=<LogBackward0>)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Low level\n",
    "torch.log(F.softmax(z, dim=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dK2BsnZcwSr7",
    "outputId": "6c895c6c-cbe4-4db3-f1a7-b24c288ecbf8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-1.3301, -1.8084, -1.6846, -1.3530, -2.0584],\n",
       "        [-1.4147, -1.8174, -1.4602, -1.6450, -1.7758],\n",
       "        [-1.5025, -1.6165, -1.4586, -1.8360, -1.6776]],\n",
       "       grad_fn=<LogSoftmaxBackward0>)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# High level\n",
    "F.log_softmax(z, dim=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Nx5wB-OqwSr8"
   },
   "source": [
    "PyTorch also has `F.nll_loss()` function that computes the negative loss likelihood."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "vIjS6qdEwSr8",
    "outputId": "5d66ec27-5b31-4186-ab63-3bbdc27f52fb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<MeanBackward0>)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Low level\n",
    "(y_one_hot * -torch.log(F.softmax(z, dim=1))).sum(dim=1).mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_T1qBNitwSr9",
    "outputId": "f4f364e9-a10f-4306-ad41-65f78b4223d8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<NllLossBackward0>)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# High level\n",
    "F.nll_loss(F.log_softmax(z, dim=1), y.long())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "alTLvMtBwSr-"
   },
   "source": [
    "PyTorch also has `F.cross_entropy` that combines `F.log_softmax()` and `F.nll_loss()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "WAvK10K0wSr_",
    "outputId": "bc4a2a23-bc09-4eea-85e4-dfd180c9fcbf"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(1.4689, grad_fn=<NllLossBackward0>)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "F.cross_entropy(z, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "V8FG-7mGwSr_"
   },
   "source": [
    "## Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NZMnCX4RyPZI"
   },
   "outputs": [],
   "source": [
    "from google.colab import drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "pjUThcSSyRgk",
    "outputId": "d46eacc8-fc1a-40b8-ecb3-1fe5efa69b36"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mounted at /content/drive\n"
     ]
    }
   ],
   "source": [
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "cTu-mKsswSsA"
   },
   "outputs": [],
   "source": [
    "xy = np.loadtxt('\"C:/Users/xhae0/Downloads/data-04-zoo.csv\", delimiter=',', dtype=np.float32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "PllSllG1wSsB"
   },
   "outputs": [],
   "source": [
    "x_train = torch.FloatTensor(xy[:, 0:-1])\n",
    "y_train = torch.LongTensor(xy[:, [-1]]).squeeze()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Q2Jx9KpEwSsB",
    "outputId": "d203850e-f9a3-4ada-ae31-70b105b4aebd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([101, 16])\n",
      "101\n",
      "tensor([[1., 0., 0., 1., 0., 0., 1., 1., 1., 1., 0., 0., 4., 0., 0., 1.],\n",
      "        [1., 0., 0., 1., 0., 0., 0., 1., 1., 1., 0., 0., 4., 1., 0., 1.],\n",
      "        [0., 0., 1., 0., 0., 1., 1., 1., 1., 0., 0., 1., 0., 1., 0., 0.],\n",
      "        [1., 0., 0., 1., 0., 0., 1., 1., 1., 1., 0., 0., 4., 0., 0., 1.],\n",
      "        [1., 0., 0., 1., 0., 0., 1., 1., 1., 1., 0., 0., 4., 1., 0., 1.]])\n"
     ]
    }
   ],
   "source": [
    "print(x_train.shape) # x_train shape\n",
    "print(len(x_train))  # x_train 길이\n",
    "print(x_train[:5])   # 첫 다섯 개"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "lMe_PQCvwSsC",
    "outputId": "ee67d9d1-7344-4eaf-c00d-a7a6483751e8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "torch.Size([101])\n",
      "101\n",
      "tensor([0, 0, 3, 0, 0])\n"
     ]
    }
   ],
   "source": [
    "print(y_train.shape) # y_train shape\n",
    "print(len(y_train))  # y_train 길이\n",
    "print(y_train[:5])   # 첫 다섯 개"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9u9RHCWzwSsD"
   },
   "outputs": [],
   "source": [
    "nb_classes = 7 # csv 파일을 보시면 0~6까지 총 7개의 class인 것을 확인할 수 있습니다!\n",
    "y_one_hot = torch.zeros((len(y_train), nb_classes))\n",
    "y_one_hot = y_one_hot.scatter(1, y_train.unsqueeze(1), 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lBk37zRZwSsE"
   },
   "source": [
    "## Training with `F.cross_entropy`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "dZ8-j1gDwSsE",
    "outputId": "45c0a594-ebc4-4548-d00a-f1bcc9f247d4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 1.945909\n",
      "Epoch  100/1000 Cost: 0.471836\n",
      "Epoch  200/1000 Cost: 0.326327\n",
      "Epoch  300/1000 Cost: 0.257839\n",
      "Epoch  400/1000 Cost: 0.215762\n",
      "Epoch  500/1000 Cost: 0.186603\n",
      "Epoch  600/1000 Cost: 0.164898\n",
      "Epoch  700/1000 Cost: 0.147955\n",
      "Epoch  800/1000 Cost: 0.134279\n",
      "Epoch  900/1000 Cost: 0.122962\n",
      "Epoch 1000/1000 Cost: 0.113422\n"
     ]
    }
   ],
   "source": [
    "# 모델 초기화\n",
    "W = torch.zeros((16,7), requires_grad = True) #가중치 W를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "b = torch.zeros(1, requires_grad = True) #편향 b를 0으로 초기화하고 학습을 통해 값이 변경될 수 있도록 하세요.\n",
    "# optimizer 설정\n",
    "optimizer = optim.SGD([W,b], lr = 0.1) #SGD optimizer를 사용하고 learning rate는 0.1로 적용하세요.\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # Cost 계산 (2)\n",
    "    z = x_train.matmul(W) + b # or .mm or @\n",
    "    cost = F.cross_entropy(z, y_train)# #PyTorch에서 제공하는 F.cross_entropy함수를 사용해 cost를 구현해보세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad() # gradient를 0으로 초기화\n",
    "    cost.backward() #  # 비용 함수를 미분하여 gradient 계산\n",
    "    optimizer.step()  # W와 b를 업데이트\n",
    "\n",
    "    # 100번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QXKz-AK8wSsE"
   },
   "source": [
    "## High-level Implementation with `nn.Module`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "LEXTvNAfwSsF"
   },
   "outputs": [],
   "source": [
    "class SoftmaxClassifierModel(nn.Module):\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(16,7) #x_train의 shape과 y_train의 class의 갯수를 고려해보세요.\n",
    "    def forward(self, x):\n",
    "        return self.linear(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mR30crMNwSsF"
   },
   "outputs": [],
   "source": [
    "model = SoftMaxClassfierModel() #SoftmaxClassifierModel()로 모델을 초기화하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "w2MRaDzJwSsF",
    "outputId": "57535a40-b203-47d9-95fe-fc6e98c8449c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch    0/1000 Cost: 1.919160\n",
      "Epoch  100/1000 Cost: 0.468405\n",
      "Epoch  200/1000 Cost: 0.320585\n",
      "Epoch  300/1000 Cost: 0.248953\n",
      "Epoch  400/1000 Cost: 0.204819\n",
      "Epoch  500/1000 Cost: 0.174506\n",
      "Epoch  600/1000 Cost: 0.152248\n",
      "Epoch  700/1000 Cost: 0.135139\n",
      "Epoch  800/1000 Cost: 0.121543\n",
      "Epoch  900/1000 Cost: 0.110461\n",
      "Epoch 1000/1000 Cost: 0.101245\n"
     ]
    }
   ],
   "source": [
    "# optimizer 설정\n",
    "optimizer = optim.SGD(model.parameters() , lr = 0.1) #SGD optimizer를 사용하고 learning rate는 0.1로 적용하세요.\n",
    "\n",
    "nb_epochs = 1000\n",
    "for epoch in range(nb_epochs + 1):\n",
    "\n",
    "    # H(x) 계산\n",
    "    prediction = model(x_train)#모델을 생성해서 예측값을 구해보세요.\n",
    "\n",
    "    # cost 계산\n",
    "    cost = F.cross_entropy(prediction, y_train) #PyTorch에서 제공하는 F.cross_entropy함수를 사용해 cost를 구현해보세요.\n",
    "\n",
    "    # cost로 H(x) 개선\n",
    "    optimizer.zero_grad() # gradient를 0으로 초기화\n",
    "    cost.backward()  # 비용 함수를 미분하여 gradient 계산\n",
    "    optimizer.step() # W와 b를 업데이트\n",
    "    \n",
    "    # 20번마다 로그 출력\n",
    "    if epoch % 100 == 0:\n",
    "        print('Epoch {:4d}/{} Cost: {:.6f}'.format(\n",
    "            epoch, nb_epochs, cost.item()\n",
    "        ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "A3Z7OL8ywSsG"
   },
   "source": [
    "<div class=\"alert alert-warning\">\n",
    "    Should I display how many it got correct in the training set?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HIF0jwKCzhOx"
   },
   "source": [
    "Q1. 각 Regression에 맞는 Hypothesis와 cost function을 적어주세요.\n",
    "\n",
    "Linear Regression : x_train * W + b, torch.mean((hypothesis-y_train) ** 2)\n",
    "\n",
    "Logistic Regression : torch.sigmoid(x_train.matmul(W)+b)-(y_train*torch.log(hipothesis)+(1-y_train)*toch.log(1-hypothesis) + (1-y_train)*torch.log(1-hypothesis)).mean()\n",
    "\n",
    "Multi Logistic regression : ?? , ??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yx3YL2aC1bYh"
   },
   "source": [
    "Q2. Lec06-2의 Logistic cost VS Cross entropy에서 두 함수가 같은 이유를 서술해주세요.\n",
    "\n",
    "=> logistic cost는 y가 0과 1일 때를 구분해주는데, 이를 행렬로 바꾸면 cross entorpy와 같은 형태기에 logistic cost는 cross entropy의 한 형태이다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Lab06_2.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}