LinearRegression/linearRegression.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": 3
  },
  "orig_nbformat": 2
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "## 线性回归\n",
    "### 模型\n",
    "假设有 $n$ 条数据, $m$ 个属性. 线性回归模型是通过参数 $\\{w_i\\}$ 和属性 $\\{ x_{ij} \\}$ 的**线性**组合来「拟合(预测)」标签列\n",
    "$$\n",
    "y_i = w_1x_{i1} + \\cdots + w_mx_{im} + b = w^Tx_i + b \\quad i = 1, 2, \\cdots, n\n",
    "$$\n",
    "其中，$w, x_i \\in \\mathbb{R}^m$\n",
    "> *举例：房价预测，销量预测，票房预测*\n",
    "\n",
    "### 求解\n",
    "> 参数估计 $w, b$\n",
    "\n",
    "#### 损失函数\n",
    "- 均方误差 MSE\n",
    "$$\n",
    "L = \\frac{1}{2n}\\sum_{i=1}^n(y_i - \\hat{y_i})^2 = \\frac{1}{2n}\\sum_{i=1}^n(w^Tx_i + b - \\hat{y_i})^2\n",
    "$$\n",
    "- 平均绝对误差 MAE\n",
    "$$\n",
    "L = \\frac{1}{n}\\sum_{i=1}^n \\bigl\\lvert y_i - \\hat{y_i} \\bigr\\rvert = \\frac{1}{n}\\sum_{i=1}^n \\bigl\\lvert w^Tx_i + b - \\hat{y_i} \\bigr\\rvert\n",
    "$$\n",
    "\n",
    "**MSE VS MAE**\n",
    "- MSE: 计算的是样本点的预测值与真实值的**欧式距离**\n",
    "  - 使用梯度下降更新参数时，越接近收敛点，导数越小\n",
    "  - 对误差大的点“惩罚”相对更大，所以，存在异常值时，对结果影响较大\n",
    "  \n",
    "- MAE: 计算的是样本点的预测值与真实值的**绝对差**\n",
    "  - 绝对值计算导数比平方麻烦，而且导数值(迭代步长)不变，计算效果差\n",
    "  - 优点是受离群异常值影响小\n",
    "\n",
    "#### 求解\n",
    "最小化损失函数\n",
    "$$\n",
    "\\min_{w, b}~L = \\min_{w,b}~\\frac{1}{2n}\\sum_{i=1}^n(w^Tx_i + b - \\hat{y_i})^2\n",
    "$$\n",
    "- 微积分法\n",
    "  $$\n",
    "  \\begin{aligned}\n",
    "  \\frac{\\partial L}{\\partial w_j} &= \\frac{1}{n}\\sum_{i=1}^nx_{ij}(w^Tx_i + b - \\hat{y_i})\\\\\n",
    "  \\frac{\\partial L}{\\partial b} &= b + \\frac{1}{n}\\sum_{i=1}^n(w^Tx_i - \\hat{y_i})\n",
    "  \\end{aligned}\n",
    "  $$\n",
    "  其中，$x_{ij}$ 表示第 $i$ 个样本的第 $j$ 个属性\n",
    "- 线性代数法  \n",
    "  最小二乘法，$A^TA x = A^Tb$\n",
    "- 机器学习  \n",
    "  利用**梯度下降**更新参数\n",
    "  $$\n",
    "  w^{n+1} = w^n - \\alpha \\nabla w^n\n",
    "  $$\n",
    "\n",
    "### 正则化\n",
    "尽可能采用「简单」的模型，可以有效提高泛化预测精度。如果模型过于「复杂」，变量值稍微有点变动，就会引起预测精度问题。\n",
    "> 正则化之所以有效，就是因为其限制参数搜索空间，降低了特征的权重，使得模型更为简单。（趋向于 0）\n",
    "\n",
    "- L1 正则化  \n",
    "  LASSO 回归，w 服从**零均值拉普拉斯分布**\n",
    "  所有权重 w 参数的绝对值之和逼迫更多 w 为零，也就是变稀疏。\n",
    "  > 实现特征的自动选择。\n",
    "- L2 正则化  \n",
    "  Ridge 回归，w 服从**零均值正态分布**\n",
    "  增加所有权重 w 参数的平方之和，逼迫所有 w 尽可能趋向零但不为零。\n",
    "  > 使模型简单\n",
    "\n",
    "在用线性回归模型拟合数据之前，首先要求数据应符合或近似符合正态分布，否则得到的拟合函数不正确。"
   ],
   "cell_type": "markdown",
   "metadata": {}
  }
 ]
}