14_Principal_Componenet_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Principal Componenet Analysis\n",
    "\n",
    "- PCA is a process of figuring out most important features or principal components that has the most impact on the target variable.\n",
    "- A techique used in ML to reduce dimension.\n",
    "\n",
    "**Exmples**\n",
    "- lets say you are building a ML model to predict house \n",
    "prices from data with features(town, area, bathroom, plot, trees nearby, price). The featurs like \"bathroom\" and \"trees nearby\" effect the price very less or nothing.\n",
    "- In problem of detecting numbers from 8x8 pixel images, there are 64 pixels in an image and some edge and corner pixels are always black and are not important feature.\n",
    "- In real life model you can have hundereds of features in data. Thus, it is necessary to detect the important ones.\n",
    "\n",
    "**What if we get rid of non important features?**\n",
    "- Faster training and inference.\n",
    "- Data visualtzation becomes easier.\n",
    "\n",
    "**How it works?**\n",
    "- we have to elminate the feature with very low variance. Like pixel_0, pixel_1, pixel_64 mostly have value = 0.0 for almost every number. Thus the variace of this feature is very low.\n",
    "- PCA(n_components=10) will give the 10 features with most  in value, that are the central pixels.\n",
    "\n",
    "**Few this to keep in mind before using PCA:**\n",
    "- Scale features before applying PCA\n",
    "- accuracy might drop\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.datasets import load_digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['data', 'target', 'frame', 'feature_names', 'target_names', 'images', 'DESCR'])"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset = load_digits()\n",
    "dataset.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pixel_0_0</th>\n",
       "      <th>pixel_0_1</th>\n",
       "      <th>pixel_0_2</th>\n",
       "      <th>pixel_0_3</th>\n",
       "      <th>pixel_0_4</th>\n",
       "      <th>pixel_0_5</th>\n",
       "      <th>pixel_0_6</th>\n",
       "      <th>pixel_0_7</th>\n",
       "      <th>pixel_1_0</th>\n",
       "      <th>pixel_1_1</th>\n",
       "      <th>...</th>\n",
       "      <th>pixel_6_6</th>\n",
       "      <th>pixel_6_7</th>\n",
       "      <th>pixel_7_0</th>\n",
       "      <th>pixel_7_1</th>\n",
       "      <th>pixel_7_2</th>\n",
       "      <th>pixel_7_3</th>\n",
       "      <th>pixel_7_4</th>\n",
       "      <th>pixel_7_5</th>\n",
       "      <th>pixel_7_6</th>\n",
       "      <th>pixel_7_7</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>...</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1792</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>...</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1793</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1794</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1795</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1796</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>8.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>8.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>14.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1797 rows × 64 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      pixel_0_0  pixel_0_1  pixel_0_2  pixel_0_3  pixel_0_4  pixel_0_5  \\\n",
       "0           0.0        0.0        5.0       13.0        9.0        1.0   \n",
       "1           0.0        0.0        0.0       12.0       13.0        5.0   \n",
       "2           0.0        0.0        0.0        4.0       15.0       12.0   \n",
       "3           0.0        0.0        7.0       15.0       13.0        1.0   \n",
       "4           0.0        0.0        0.0        1.0       11.0        0.0   \n",
       "...         ...        ...        ...        ...        ...        ...   \n",
       "1792        0.0        0.0        4.0       10.0       13.0        6.0   \n",
       "1793        0.0        0.0        6.0       16.0       13.0       11.0   \n",
       "1794        0.0        0.0        1.0       11.0       15.0        1.0   \n",
       "1795        0.0        0.0        2.0       10.0        7.0        0.0   \n",
       "1796        0.0        0.0       10.0       14.0        8.0        1.0   \n",
       "\n",
       "      pixel_0_6  pixel_0_7  pixel_1_0  pixel_1_1  ...  pixel_6_6  pixel_6_7  \\\n",
       "0           0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "1           0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "2           0.0        0.0        0.0        0.0  ...        5.0        0.0   \n",
       "3           0.0        0.0        0.0        8.0  ...        9.0        0.0   \n",
       "4           0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "...         ...        ...        ...        ...  ...        ...        ...   \n",
       "1792        0.0        0.0        0.0        1.0  ...        4.0        0.0   \n",
       "1793        1.0        0.0        0.0        0.0  ...        1.0        0.0   \n",
       "1794        0.0        0.0        0.0        0.0  ...        0.0        0.0   \n",
       "1795        0.0        0.0        0.0        0.0  ...        2.0        0.0   \n",
       "1796        0.0        0.0        0.0        2.0  ...        8.0        0.0   \n",
       "\n",
       "      pixel_7_0  pixel_7_1  pixel_7_2  pixel_7_3  pixel_7_4  pixel_7_5  \\\n",
       "0           0.0        0.0        6.0       13.0       10.0        0.0   \n",
       "1           0.0        0.0        0.0       11.0       16.0       10.0   \n",
       "2           0.0        0.0        0.0        3.0       11.0       16.0   \n",
       "3           0.0        0.0        7.0       13.0       13.0        9.0   \n",
       "4           0.0        0.0        0.0        2.0       16.0        4.0   \n",
       "...         ...        ...        ...        ...        ...        ...   \n",
       "1792        0.0        0.0        2.0       14.0       15.0        9.0   \n",
       "1793        0.0        0.0        6.0       16.0       14.0        6.0   \n",
       "1794        0.0        0.0        2.0        9.0       13.0        6.0   \n",
       "1795        0.0        0.0        5.0       12.0       16.0       12.0   \n",
       "1796        0.0        1.0        8.0       12.0       14.0       12.0   \n",
       "\n",
       "      pixel_7_6  pixel_7_7  \n",
       "0           0.0        0.0  \n",
       "1           0.0        0.0  \n",
       "2           9.0        0.0  \n",
       "3           0.0        0.0  \n",
       "4           0.0        0.0  \n",
       "...         ...        ...  \n",
       "1792        0.0        0.0  \n",
       "1793        0.0        0.0  \n",
       "1794        0.0        0.0  \n",
       "1795        0.0        0.0  \n",
       "1796        1.0        0.0  \n",
       "\n",
       "[1797 rows x 64 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(dataset.data, columns=dataset.feature_names)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x = df\n",
    "y = dataset.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.        , -0.33501649, -0.04308102, ..., -1.14664746,\n",
       "        -0.5056698 , -0.19600752],\n",
       "       [ 0.        , -0.33501649, -1.09493684, ...,  0.54856067,\n",
       "        -0.5056698 , -0.19600752],\n",
       "       [ 0.        , -0.33501649, -1.09493684, ...,  1.56568555,\n",
       "         1.6951369 , -0.19600752],\n",
       "       ...,\n",
       "       [ 0.        , -0.33501649, -0.88456568, ..., -0.12952258,\n",
       "        -0.5056698 , -0.19600752],\n",
       "       [ 0.        , -0.33501649, -0.67419451, ...,  0.8876023 ,\n",
       "        -0.5056698 , -0.19600752],\n",
       "       [ 0.        , -0.33501649,  1.00877481, ...,  0.8876023 ,\n",
       "        -0.26113572, -0.19600752]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler()\n",
    "x_scaled = scaler.fit_transform(x)\n",
    "x_scaled"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(x_scaled, y, test_size=0.2, random_state=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9722222222222222"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train,Y_train)\n",
    "model.score(X_test,Y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1797, 29)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA\n",
    "\n",
    "pca = PCA(0.95) # retain 95% of usefull features\n",
    "\n",
    "x_pca = pca.fit_transform(x)\n",
    "x_pca.shape # 29 features left"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_pca, X_test_pca, Y_train, Y_test = train_test_split(x_pca, y, test_size=0.2, random_state=30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "c:\\Users\\utkarsh.alpuria\\Desktop\\My Work\\Python\\Python ML\\Python ML Algorithm\\.ml_algo_py_venv\\lib\\site-packages\\sklearn\\linear_model\\_logistic.py:469: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9722222222222222"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LogisticRegression()\n",
    "model.fit(X_train_pca, Y_train)\n",
    "model.score(X_test_pca, Y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "the accuracy is comparmised a little but the computation required was very less."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".ml_algo_py_venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}