7.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# 7.1 参数的点估计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### 引言"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "上一章, 我们介绍了总体、样本、简单随机样本、统计量和抽样分布的概念, 介绍了统计中常用的三大分布, 给出了几个重要的抽样分布定理. 它们是进一步学习统计推断的基础."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "![](./figures/71.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "研究统计量的性质和评价一个统计推断的优良性, 完全取决于其抽样分布的性质."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](./figures/72.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**参数估计**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**现在我们来介绍一类重要的统计推断问题**\n",
    "\n",
    "在参数估计问题中, 假定总体分布形式已知, 未知的仅仅是一个或几个参数."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "![](./figures/74.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 参数估计的一般提法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "设有一个统计总体, 总体的分布函数为 $F(x,\\theta)$, 其中 $\\theta$ 为未知参数($\\theta$ 可以是向量). 现从该总体抽样, 得样本\n",
    "\n",
    "$$\n",
    "X_1,X_2,\\cdots,X_n\n",
    "$$\n",
    "\n",
    "要依据该样本对参数 $\\theta$ 作出估计, 或估计 $\\theta$ 的某个已知函数 $g(\\theta)$ .这类问题称为**参数估计**.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**点估计**\n",
    "\n",
    "    ———— 由随机抽样估计参数的具体值的方法.\n",
    "    \n",
    "**区间估计**\n",
    "\n",
    "    ———— 根据置信度估计未知参数的取值范围."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "假如我们要估计某队男生的平均身高. 假定身高服从正态分布 $N(\\mu, 0.1^2)$ .\n",
    "现从该总体选取容量为 $5$ 的样本, 我们的任务是要根据选出的样本( $5$ 个数)求出总体均值 $\\mu$ 的估计. 而全部信息就由这 $5$ 个数组成 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "设这 $5$ 个数为: $1.65,1.67,1.68,1.78,1.69$\n",
    "\n",
    "估计 $\\mu$ 为 $1.68$, 这是**点估计**.\n",
    "\n",
    "估计 $\\mu$ 在区间 $[1.57,1.84]$ 内, 这是**区间估计**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 一、衡量估计量好坏的标准"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "参数估计需要讨论以下几个问题:\n",
    "1. 我们希望一个“好的”估计量具有什么特性?\n",
    "1. 怎样决定一个估计量是否比另一个估计量“好”？\n",
    "1. 如何求得合理的估计量？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "常用的几条标准是：\n",
    "1. 无偏性\n",
    "1. 有效性\n",
    "1. 一致性\n",
    "\n",
    "这里我们重点介绍前面两个标准 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**1.无偏性**\n",
    "估计量是随机变量, 对于不同的样本值会得到不同的估计值. 我们希望估计值在未知参数真值附近摆动, 而它的期望值等于未知参数的真值. 这就导致无偏性这个标准 . \n",
    "\n",
    "设 $\\widehat \\theta(X_1,X_2,\\dots,X_n)$ 是未知参数 $\\theta$ 的估计量, 若\n",
    "\n",
    "$$E(\\widehat \\theta)=\\theta$$\n",
    "\n",
    "则称 $\\widehat \\theta$ 为 $\\theta$ 的无偏估计 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例1** 设总体 $X$ 具有均匀分布, 其密度函数为\n",
    "\n",
    "$$\n",
    "f(x,\\theta)=\n",
    "\\begin{cases}\n",
    "\\frac{1}{\\theta} & 0<x\\leq \\theta,0<\\theta<\\infty\\\\\n",
    "0&其他\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "求 $\\theta$ 的无偏估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:** 总体 $X$ 的均值 $E(X)=\\int_{0}^{\\theta}x\\frac{1}{\\theta}\\mathrm{d}x=\\frac{1}{\\theta}\\frac{x^2}{2}|_{0}^{\\theta}=\\frac{\\theta}{2}$\n",
    "\n",
    "又 $E(\\overline X)=E(X)=\\frac{\\theta}{2}$ ,取 $\\widehat \\theta=2\\overline X$,知 \n",
    "\n",
    "$$\n",
    "E(\\widehat \\theta)=2E(\\overline X)=2E(X)=2\\times\\frac{\\theta}{2}=\\theta\n",
    "$$\n",
    "\n",
    "所以求得 $\\widehat \\theta=2\\overline X$ 是 $\\theta$ 无偏估计量."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例2**  设总体 $X$ 的 $k$ 阶矩 $E(X^k)$ 存在,证明样本的 $k$ 阶矩是 $E(X^k)$ 的无偏估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**证明:** 因为\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "E(\\frac{1}{n}\\sum_{i=1}^{n}X_{i}^{k})=\\frac{1}{n}E(\\sum_{i=1}^{n}X_{i}^{k})=\\frac{1}{n}\\sum_{i=1}^{n}E(X_{i}^{k})\\\\\n",
    "= \\frac{1}{n}\\sum_{i=1}^{n}E(X^{k})\n",
    "= E(X^{k})\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "所以样本的 $k$ 阶矩是 $E(X^{k})$ 的无偏估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例3**  设总体的方差 $D(X)$ 存在,试证样本二阶中心矩 $B_2$ 是总体方差 $D(X)$ 的有偏估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**证明:**\n",
    "$$\n",
    "\\begin{align}\n",
    "E(B_2)=&E[\\frac{1}{n}\\sum_{i=1}^{n}(X_{i}-\\overline X)^2]\\\\\n",
    "=&E[\\frac{1}{n}\\sum_{i=1}^{n}X_{i}^{2}-\\overline X^2]\\\\\n",
    "=&E(\\frac{1}{n}\\sum_{i=1}^{n}X_{i}^{2})-E(\\overline X^2)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\n",
    "\\begin{align}\n",
    "=&E(X^2)-E(\\overline X^2)\\\\\n",
    "=&D(X)+[E(X)]^2-D(\\overline X)-[E(\\overline X)]^2\\\\\n",
    "=&D(X)-\\frac{1}{n}D(X)\\\\\n",
    "=&\\frac{n-1}{n}D(X)\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "所以 $B_2$ 是总体方差 $D(X)$ 的有偏估计.\n",
    "\n",
    "**注:** $S^2=\\frac{1}{n-1}\\sum_{i=1}^{n}(X_i-\\overline X)^2$ 是 $D(X)$ 的无偏估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例4**  设总体 $X$ 的均值为 $E(X)=\\mu$, 方差为 $D(X)=\\sigma^2$, 证明:\n",
    "\n",
    "(1) 样本均值 $\\overline X =\\frac{1}{n}\\sum_{i=1}^{n}X_i$, 是总体均值 $\\mu$ 的无偏估计量.\n",
    "\n",
    "(2) 样本方差 $\\overline {S^2} =\\frac{1}{n-1}\\sum_{i=1}^{n}(X_i-\\overline X)^2$, 是总体方差 $\\sigma^2$ 的无偏估计量."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**证:** 因为样本 $X_1,X_2,\\dots,X_n$, 相互独立, 且与总体 $X$ 服从相同分布, 所以有\n",
    "\n",
    "$$\n",
    "E(X_i) = \\mu,\\qquad D(X_i) = \\sigma^2,\\qquad i= 1,2,\\dots,n\n",
    "$$\n",
    "\n",
    "由数学期望与方差的性质可知\n",
    "\n",
    "$$\n",
    "E(\\overline X)=E(\\frac{1}{n}\\sum_{i=1}^{n}X_i)=\\frac{1}{n}E(\\sum_{i=1}^{n}X_i)=\\frac{1}{n}\\sum_{i=1}^{n}E(X_i)=\\mu\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "所以 $\\overline X$ 是 $\\mu$ 的无偏估计量. 所以\n",
    "\n",
    "$$\n",
    "\\widehat{\\mu} = \\overline X\n",
    "$$\n",
    "\n",
    "(2) \n",
    "$$\n",
    "S^2 =\\frac{1}{n-1}\\sum_{i=1}^{n}(X_i-\\overline X)^2 =\\frac{1}{n-1}(\\sum_{i=1}^{n}X_i^2-n\\overline X^2)\n",
    "$$\n",
    "\n",
    "而\n",
    "$$\n",
    "E(X_i^2)=D(X_i)+[E(X_i)]^2 =\\sigma^2+\\mu^2,i = 1,2,\\dots,n\\\\\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\\begin{align}\n",
    "E(\\overline X^2)\n",
    "&= D(\\overline X)+[E(\\overline X)]^2\\\\\n",
    "&=D(\\frac{1}{n}\\sum_{i=1}^{n}X_i)+\\mu^2\\\\\n",
    "&=\\frac{1}{n^2}\\sum_{i=1}^{n}D(X_i)+\\mu^2\\\\\n",
    "&=\\frac{1}{n^2} n\\sigma^2+\\mu^2\\\\\n",
    "&=\\frac{\\sigma^2}{n}+\\mu^2\\\\\n",
    "\\end{align}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "由此可得\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "E(S^2) &= E[\\frac{1}{n-1}(\\sum_{i=1}^{n}X_i^{2}-n\\overline X^2)]\\\\\n",
    "&=\\frac{1}{n-1}[n(\\sigma^2+\\mu^2)-n(\\frac{\\sigma^2}{n}+\\mu^2)]\\\\\n",
    "&=\\sigma^2\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "所以 $S^2$ 是 $\\sigma^2$ 的无偏估计量:\n",
    "\n",
    "$$\n",
    "\\widehat\\sigma^2=S^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "无偏性是对估计量的一个常见而重要的要求 .\n",
    "\n",
    "无偏性的实际意义是指没有系统性的偏差 .\n",
    "\n",
    "例如, 用样本均值作为总体均值的估计时, 虽无法说明一次估计所产生的偏差, 但这种偏差随机地在 $0$ 的周围波动, 对同一统计问题大量重复使用不会产生系统偏差 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "一个参数往往有不止一个无偏估计.\n",
    "\n",
    "若 $\\widehat \\theta_1$ 和 $\\widehat\\theta_2$ 都是参数 $\\theta$ 的无偏估计量，我们可以比较 $E(\\widehat\\theta_1-\\theta)^2$ 和 $E(\\widehat\\theta_2-\\theta)^2$ 的大小来决定二者谁更优. 由于 \n",
    "\n",
    "$$\n",
    "D(\\widehat\\theta_1)=E(\\widehat\\theta_1-\\theta)^2\\\\\n",
    "D(\\widehat\\theta_2)=E(\\widehat\\theta_2-\\theta)^2\n",
    "$$\n",
    "\n",
    "所以无偏估计以方差小者为好, 这就引进了有效性这一概念."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**2.有效性**\n",
    "\n",
    "设\n",
    "$$\n",
    "\\widehat\\theta_1=\\widehat\\theta_1(X_1,\\dots,X_n)\n",
    "$$\n",
    "\n",
    "和\n",
    "\n",
    "$$\n",
    "\\widehat\\theta_2=\\widehat\\theta_2(X_1,\\dots,X_n)\n",
    "$$\n",
    "\n",
    "都是参数 $\\theta$ 的无偏估计量, 若有\n",
    "\n",
    "$$\n",
    "D(\\widehat\\theta_1)<D(\\widehat\\theta_2)\n",
    "$$\n",
    "\n",
    "则称 $\\theta_1$ 较 $\\theta_2$ 有效."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例5** 设 $(X_1,X_2,X_3)$ 是来自总体 $X$ 的一个样本, 试证下面三个估计量都是总体均值 $E(X)$ 的无偏估计量.\n",
    "\n",
    "$$\n",
    "\\widehat \\mu_1=\\overline X;\\widehat \\mu_2=\\frac{1}{2}X_1+\\frac{1}{3}X_2+\\frac{1}{6}X_3;\\widehat \\mu_3=X_1\n",
    "$$\n",
    "\n",
    "且 $\\widehat\\mu_1$ 比 $\\widehat\\mu_2,\\widehat\\mu_3$ 都有效."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**证明:**易知\n",
    "$$\n",
    "E(\\widehat\\mu_1) = E(\\widehat\\mu_2)=E(\\widehat\\mu_3)\n",
    "$$\n",
    "\n",
    "且\n",
    "$$\n",
    "D(\\widehat\\mu_1)=D(\\overline X)=D(X)/3\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\n",
    "D(\\widehat\\mu_2)=D(\\frac{1}{2}X_1+\\frac{1}{3}X_2+\\frac{1}{6}X_3)=14D(X)/36\n",
    "$$\n",
    "\n",
    "$$\n",
    "D(\\widehat\\mu_3)=D(X_1)=D(X)\n",
    "$$\n",
    "\n",
    "故有 $D(\\widehat\\mu_1)<D(\\widehat\\mu_2)<D(\\widehat\\mu_3)$, 所以 $\\widehat\\mu_1$ 比 $\\widehat\\mu_2,\\widehat\\mu_3$ 都有效."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**3.一致性**\n",
    "\n",
    "如果当样本容量 $n\\to \\infty$ 时,\n",
    "\n",
    "$$\n",
    "\\widehat\\theta=\\widehat\\theta(X_1,X_2,\\dots,X_n)\n",
    "$$\n",
    "\n",
    "按概率收敛于 $\\theta$, 即对于任意 $\\varepsilon>0$, 有\n",
    "\n",
    "$$\n",
    "\\lim_{n\\to\\infty}P(|\\widehat\\theta-\\theta|<\\varepsilon)=1\n",
    "$$\n",
    "\n",
    "则称 $\\widehat\\theta$ 为 $\\theta$ 的一致估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "在数理统计中常用到最小方差无偏估计.\n",
    "\n",
    "它的定义是: 设 $X_1,X_2,\\dots,X_n$ 是取自总体 $X$ 的一个样本,\n",
    "\n",
    "$$\n",
    "\\widehat\\theta(X_1,X_2,\\dots,X_n)\n",
    "$$\n",
    "\n",
    "是未知参数 $\\theta$ 的一个估计量, 若 $\\widehat\\theta$ 满足:\n",
    "1. $E(\\widehat\\theta)=\\theta$, 即 $\\widehat\\theta$ 为 $\\theta$ 的无偏估计;\n",
    "1. $D(\\widehat\\theta)\\leq D(\\widehat\\theta_k)$, $\\widehat\\theta_k$ 是 $\\theta$ 的任一无偏估计.\n",
    "\n",
    "则称 $\\widehat\\theta$ 为 $\\theta$ 最小方差无偏估计（也称为最佳无偏估计）."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### 二、求点估计的两种常用方法"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "1. 矩估计法\n",
    "1. 极大似然估计\n",
    "1. 最小二乘法\n",
    "1. 贝叶斯方法\n",
    "\n",
    "这里主要介绍前两种方法."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "1.矩估计法\n",
    "\n",
    "它是基于一种简单的“替换”思想建立起来的一种估计方法, 它是英国统计学家K. 皮尔逊最早提出的， 其基本思想是用样本矩估计总体矩.\n",
    "\n",
    "理论依据:大数定律或格列汶科定理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "记总体 $k$ 阶矩为 $\\mu_k=E(X^k)$, 样本 $k$ 阶矩为 \n",
    "\n",
    "$$A_k=\\frac{1}{n}\\sum_{i=1}^{n}X_{i}^{k}$$\n",
    "\n",
    "记总体 $k$ 阶中心矩为 $v_k=E[X-E(X)]^k$, 样本 $k$ 阶中心矩为 \n",
    "\n",
    "$$ B_k=\\frac{1}{n}\\sum_{i=1}^{n}（X_i-\\overline X)^k$$\n",
    "\n",
    "用相应的样本矩去估计总体矩的估计方法就称为**矩估计法**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "设总体的分布函数中含有 $k$ 个未知参数 $\\theta_1,\\dots,\\theta_k$, 那么它的前 $k$ 阶矩 $\\mu_1,\\dots,\\mu_k$, 一般都是这 $k$ 个参数的函数, 记为:\n",
    "\n",
    "$$\n",
    "\\mu_i = g_i(\\theta_1,\\dots,\\theta_k) ,\\qquad i=1,2,\\dots,k\n",
    "$$\n",
    "\n",
    "从这 $k$ 个方程中解出\n",
    "\n",
    "$$\n",
    "\\theta_j=h_j(\\mu_1,\\dots,\\mu_k)\\qquad j=1,2,\\dots,k\n",
    "$$\n",
    "\n",
    "那么用诸 $\\mu_i$ 的估计量 $A_i$ 分别代替上式中的诸 $\\mu_i$, 即可得诸 $\\theta_j$ 的矩估计量:\n",
    "\n",
    "$$\n",
    "\\widehat \\theta_j = h_j(A_1,\\dots,A_k)\\qquad j=1,2,\\dots,k\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "设\n",
    "$$\n",
    "\\begin{cases}\n",
    "E(X) = \\mu_1(\\theta_1,\\dots,\\theta_k)=A_1\\\\\n",
    "E(X^2) = \\mu_2(\\theta_1,\\dots,\\theta_k)=A_2\\\\\n",
    "\\dots\\\\\n",
    "E(X^k) = \\mu_k(\\theta_1,\\dots,\\theta_k)=A_k\\\\\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "得到含有未知参数 $(\\theta_1,\\dots,\\theta_k)$ 的 $k$ 个方程, 解这 $k$ 个联立方程组就可以得到 $(\\theta_1,\\dots,\\theta_k)$　一组解:\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "\\widehat\\theta_1 = \\widehat\\theta_1(X_1,\\dots,X_n)\\\\\n",
    "\\widehat\\theta_2 = \\widehat\\theta_2(X_1,\\dots,X_n)\\\\\n",
    "\\dots\\\\\n",
    "\\widehat\\theta_k = \\widehat\\theta_k(X_1,\\dots,X_n)\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "用上面的解来估计参数 $\\theta_j$ 就是矩法估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例６**　设总体 $X$ 的概率密度为\n",
    "\n",
    "$$\n",
    "f(x)=\n",
    "\\begin{cases}\n",
    "(\\alpha+1)x^{\\alpha},\\,0<x<1\\\\\n",
    "0&其他\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "其中 $\\alpha>-1$ 是未知参数, $X_1,X_2,\\dots,X_n$ 是取自 $X$ 的样本, 求参数 $\\alpha$ 的矩估计.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:**\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mu_1 &= E(X)\\\\\n",
    "&=\\int_{0}^{1}x(\\alpha+1)x^{\\alpha}\\mathrm{d}x\\\\\n",
    "&=(\\alpha+1)\\int_{0}^{1}x^{\\alpha+1}\\mathrm{d}x\\\\\n",
    "&=\\frac{\\alpha+1}{\\alpha+2}\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "由矩法，\n",
    "$$\n",
    "\\overline X=\\frac{\\alpha+1}{\\alpha+2}\n",
    "$$\n",
    "\n",
    "从中解得\n",
    "\n",
    "$$\n",
    "\\widehat \\alpha =\\frac{2\\overline X-1}{1-\\overline X}\n",
    "$$\n",
    "\n",
    "即 $\\alpha$ 的矩估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例７**　设 $X_1,X_2,\\dots,X_n$ 是取自总体 $X～P(\\lambda)$ 的一个样本, 求 $\\lambda$ 的矩估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:** 总体 $X$ 的期望为　$E(X) = \\lambda$, 从而得出方程\n",
    "\n",
    "$$\n",
    "\\lambda = \\frac{1}{n}\\sum_{i=1}^{n}X_i\n",
    "$$\n",
    "\n",
    "所以 $\\lambda$ 的矩估计量为\n",
    "\n",
    "$$\n",
    "\\widehat \\lambda = \\frac{1}{n}\\sum_{i=1}^{n}X_i=\\overline X\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例8**　设 $X_1,X_2,\\dots,X_n$ 是取自总体 $X～e(\\lambda)$ 的一个样本, 求 $\\lambda$ 的矩估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:**其概率密度函数为\n",
    "\n",
    "$$\n",
    "f(x,\\lambda)=\n",
    "\\begin{cases}\n",
    "\\lambda e^{-\\lambda x},&x>0\\\\\n",
    "0,&x\\leq 0\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "总体 $X$ 的期望为 $E(X)=\\int_{0}^{+\\infty}x\\lambda e^{-\\lambda x}\\mathrm{d}x=\\frac{1}{\\lambda}$\n",
    "\n",
    "从而得方程 \n",
    "\n",
    "$$\n",
    "\\frac{1}{\\lambda}=\\frac{1}{n}\\sum_{i=1}^{n}X_i\n",
    "$$\n",
    "\n",
    "所以 $\\lambda$ 的矩估计量为 $\\widehat \\lambda=\\frac{1}{\\sum_{i=1}^{n}X_i}=\\frac{1}{\\overline X}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例9**　设 $X_1,X_2,\\dots,X_n$ 是取自总体 $X$ 的一个样本,\n",
    "\n",
    "$$\n",
    "X～f(X)=\\begin{cases}\n",
    "\\frac{1}{\\theta}e^{-(x-\\mu)/\\theta},&x\\geq \\mu\\\\\n",
    "0,&\\text{ 其他}\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "其中 $\\theta,\\mu$ 为未知参数, $\\theta>0$, 求 $\\theta,\\mu$ 的矩估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:**由密度函数知，$X-\\mu$ 具有均值为 $\\theta$ 的指数分布, 故\n",
    "\n",
    "$$E(X-\\mu)=\\theta, \\,D(X-\\mu)=\\theta^2$$\n",
    "\n",
    "即 $E(X)=\\mu+\\theta$, $D(X) =\\theta^2$\n",
    "\n",
    "令\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "\\mu+\\theta=\\overline X\\\\\n",
    "\\theta^2=\\frac{1}{n}\\sum_{i=1}^{n}(X_i-\\overline X)^2\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "解得\n",
    "\n",
    "$$\n",
    "\\widehat \\mu= \\overline X - \\sqrt{\\frac{1}{n}\\sum_{i=1}^{n}(X_i-\\overline X)^2}\\\\\n",
    "\\widehat\\theta= \\sqrt{\\frac{1}{n}\\sum_{i=1}^{n}(X_i-\\overline X)^2}\n",
    "$$\n",
    "\n",
    "$\\widehat \\mu,\\widehat\\theta$ 即为参数 $\\mu,\\theta$ 的矩估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* 矩法的优点是简单易行, 并不需要事先知道总体是什么分布; \n",
    "* 缺点是当总体类型已知时, 没有充分利用分布提供的信息. \n",
    "* 一般场合下, 矩估计量不具有唯一性. 其主要原因是在于建立矩法方程时, 选取那些总体矩用相应样本矩代替带有一定的随意性."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "2.极大似然估计\n",
    "\n",
    "极大似然估计是在总体类型已知条件下, 使用的一种参数估计方法. 它首先是由德国数学家高斯在1821年提出的, 然而, 这个方法常归功于英国统计学家费歇. 费歇在1922年重新发现了这一方法, 并首先研究了这种方法的一些性质 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "**极大似然法的基本思想**\n",
    "\n",
    "先看一个简单例子: 某位同学与一位猎人一起外出打猎, 一只野兔从前方窜过, 只听一声枪响,野兔应声倒下. 如果要你推测, 是谁打中的呢? 你会如何想呢?\n",
    "\n",
    "我们猜测: 只发一枪便打中野兔, 猎人命中的概率一般大于这位同学命中的概率. 看来这一枪是猎人射中的. 这个例子所作的推断已经体现了极大似然法的基本思想."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "极大似然原理的直观想法是:\n",
    "一个随机试验如有若干个可能的结果 $A,B,C,\\dots$. 若在一次试验中, 结果 $A$ 出现, 则一般认为 $A$ 出现的概率最大, 也即试验条件对 $A$ 出现最有利. 或者说在试验的很多可能条件中, 认为应该是使事件 $A$ 发生的概率为最大的那种条件存在.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "例如: 有两罐外观相同的乒乓球, 各装 $1000$ 个乒乓球. 一罐有 $950$ 个红球, $50$ 个白球; 另一罐 $50$ 个红球, $950$ 个白球. 现任取一罐, 任取一个乒乓球, 发现取得红球.问所取乒乓球来自哪一罐?\n",
    "\n",
    "答:来自 $950$ 个红球, $50$ 个白球的那一罐."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例10** 设 $X～B(1,p)$, $p$ 未知. 设想我们事先知道 $p$ 只有两种可能: $p=0.7$ 或 $p=0.3$, 如今重复试验三次得 $0,0,0$, 问应如何估计 $p$?\n",
    "\n",
    "由概率论的知识, $3$ 次试验中出现 $1$ 的次数\n",
    "\n",
    "$$\n",
    "Y～B(3,p)\\\\\n",
    "P(Y=k)=\\dbinom{3}{k}p^k(1-p)^{3-k}\\qquad k=0,1,2,3\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "如果有 $p_1,p_2,\\dots,p_m$ 可供选择, 又如何合理地选 $p$ 呢? 若重复进行试验 $n$ 次,结果 “$1$” 出现 $k$ 次($0\\leq k\\leq n$),我们计算一切可能的\n",
    "\n",
    "$$\n",
    "P(Y=k;p_i)=Q_i\\qquad i=1,2,\\dots,m\n",
    "$$\n",
    "\n",
    "从中选取使 $Q_i$ 最大的 $p_i$ 作为 $p$ 的估计. 比方说, 当\n",
    "\n",
    "$$\n",
    "p=p_{i_0}\n",
    "$$\n",
    "\n",
    "时 $Q_i$ 最大,\n",
    "\n",
    "$$\n",
    "P(Y=k;p_{i_0})\\geq P(Y=k;p_i)\\qquad i=1,2,\\dots,m\n",
    "$$\n",
    "\n",
    "则估计参数 $p$ 为 $\\widehat p =p_{i_0}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "如果只知道 $0<p<1$, 并且实测记录是\n",
    "\n",
    "$$\n",
    "Y=k (0\\leq k\\leq n),\n",
    "$$\n",
    "\n",
    "又应如何估计 $p$ 呢?注意到\n",
    "\n",
    "$$\n",
    "P(Y=k;p)=\\dbinom{n}{k}p^k(1-p)^{n-k}=f(p)\n",
    "$$\n",
    "\n",
    "是 $p$ 的函数, 可用求导的方法找到 $f(p)$ 的极大值点 $p$. 但因 $f(p)$ 与 $\\ln f (p)$ 达到极大值的自变量相同, 故问题可转化为求 $\\ln f(p)$ 的极大值点."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\n",
    "\\ln f(p) = \\ln\\dbinom{n}{k}+k\\ln p+(n-k)\\ln(1-p)\n",
    "$$\n",
    "\n",
    "将 $\\ln f(p)$ 对 $p$ 求导并令其为 $0$,\n",
    "\n",
    "$$\n",
    "\\frac{d\\ln f(p)}{dp}=\\frac{k}{p}-\\frac{n-k}{1-p}=0\n",
    "$$\n",
    "\n",
    "可得"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\n",
    "p(n-k)=k(1-p)\n",
    "$$\n",
    "\n",
    "从中解得\n",
    "\n",
    "$$\n",
    "\\widehat p =\\frac{k}{n}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "这时，对一切 $0<p<1$, 均有\n",
    "\n",
    "$$\n",
    "P(Y=k;\\widehat p)\\geq P(Y=k;p)\n",
    "$$\n",
    "\n",
    "则估计参数 $p$ 为 $\\widehat p = \\frac{k}{n}$\n",
    "\n",
    "以上这种选择一个参数使得实验结果具有最大概率的思想就是极大似然法的基本思想 ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**定义** 设总体X的分布类型已知, 但含有未知参数 $\\theta$.\n",
    "\n",
    "(1). 设离散型总体 $X$ 具有概率分布率 $p(x,\\theta)$,则样本 $(X_1,\\dots,X_n)$ 的联合分布率为\n",
    "\n",
    "$$\n",
    "p(x_1,\\theta)p(x_2,\\theta)\\dots p(x_n,\\theta)=\\Pi_{i=1}^{n}p(x_i;\\theta)\n",
    "$$\n",
    "\n",
    "称为**似然函数**，记作\n",
    "\n",
    "$$\n",
    "L(\\theta)=L(x_1,x_2,\\dots,x_n,\\theta)=\\Pi_{i=1}^{n}p(x_i;\\theta)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "(2). 设连续型总体 $X$ 具有概率分布率 $f(x,\\theta)$,则样本 $(X_1,\\dots,X_n)$ 的联合分布函数为\n",
    "\n",
    "$$\n",
    "f(x_1,\\theta)f(x_2,\\theta)\\dots f(x_n,\\theta)=\\Pi_{i=1}^{n}f(x_i,\\theta)\n",
    "$$\n",
    "\n",
    "称为似然函数，记作\n",
    "\n",
    "$$\n",
    "L(\\theta)=L(x_1,x_2,\\dots,x_n,\\theta)=\\Pi_{i=1}^{n}f(x_i,\\theta)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "求极大似然估计(MLE)的一般步骤是：\n",
    "1. 由总体导出样本的联合概率分布 (或联合密度);\n",
    "1. 把样本联合概率分布(或联合密度)中自变量看成已知常数,而把参数 $\\theta$ 看作自变量, 得到似然函数 $L(\\theta)$;\n",
    "1. 求似然函数 $L(\\theta)$ 的最大值点, 常常转化为求 $\\ln L(\\theta)$ 的最大值点, 即 $\\theta$ 的MLE;\n",
    "1. 在最大值点的表达式中, 用样本值代入就得参数的极大似然估计值."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**两点说明:**\n",
    "1. 求似然函数 $L(\\theta)$ 的最大值点, 可以应用微积分中的技巧.\n",
    "\n",
    "1. 用上述求导方法求参数的MLE有时行不通时, 要用极大似然原则来求.\n",
    "\n",
    "下面举例说明如何求极大似然估计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例11** 设 $x_1,x_2,\\dots,x_n$ 是取自总体 $X～B(1, p)$ 的一个样本, 求参数 $p$ 的极大似然估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:**似然函数为:\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "L(p) &= L(x_1,x_2,\\dots,x_n;p)\\\\\n",
    "&=\\Pi_{i=1}^{n}p^{x_i}(1-p)^{1-x_i}\\\\\n",
    "&=p^{\\sum_{i=1}^{n}x_i}(1-p)^{n-\\sum_{i=1}^{n}x_i}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "$$\n",
    "L(p) = p^{\\sum_{i=1}^{n}x_i}(1-p)^{n-\\sum_{i=1}^{n}x_i}\n",
    "$$\n",
    "\n",
    "对数似然函数为:\n",
    "\n",
    "$$\n",
    "\\ln L(p) = \\sum_{i=1}^{n}x_i\\ln(p)+(n-\\sum_{i=1}^{n}x_i)\\ln(1-p)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "对 $p$ 求导并令其为 $0$,\n",
    "\n",
    "$$\n",
    "\\frac{d{\\ln L(p)}}{d{p}} = \\frac{1}{p}\\sum_{i=1}^{n}x_i-\\frac{1}{1-p}(n-\\sum_{i=1}^{n}x_i)=0\n",
    "$$\n",
    "\n",
    "得 $\\widehat p = \\frac{1}{n}\\sum_{i=1}^{n}x_i = \\overline x$,即为 $p$ 的 MLE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例12** 设 $x_1,x_2,\\dots,x_n$ 是取自总体 $X$ 的一个样本,\n",
    "\n",
    "$$\n",
    "X～f(x)=\n",
    "\\begin{cases}\n",
    "\\theta x^{\\theta-1}, & \\,0<x<1\\\\\n",
    "0,&\\text{ 其他}\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "其中 $\\theta>0$ ,求 $\\theta$ 的极大似然估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:** 似然函数为\n",
    "\n",
    "$$\n",
    "L(\\theta) = \\Pi_{i=1}^{n}\\theta x_{i}^{\\theta-1}=\\theta^n(\\Pi_{i=1}^{n}x_{i})^{\\theta-1}\n",
    "$$\n",
    "\n",
    "对数似然函数为\n",
    "\n",
    "$$\n",
    "\\ln L(\\theta)=n\\ln\\theta+(\\theta-1)\\sum_{i=1}^{n}\\ln x_i\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "求导并令其为 $0$,\n",
    "\n",
    "$$\n",
    "\\frac{d{\\ln L(\\theta)}}{d{\\theta}}=0\n",
    "$$\n",
    "\n",
    "从中解得\n",
    "\n",
    "$$\n",
    "\\theta = -\\frac{n}{\\sum_{i=1}^{n}\\ln x_i}\n",
    "$$\n",
    "\n",
    "即为 $\\theta$ 的 MLE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例13** 设随机变量X服从泊松分布\n",
    "\n",
    "$$\n",
    "P(X=k)=\\frac{\\lambda^ke^{-\\lambda}}{k!}\\qquad k=0,1,2,\\dots\n",
    "$$\n",
    "\n",
    "其中 $\\lambda>0$ 是一未知参数，求 $\\lambda$ 的极大似然估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:** 设 $(X_1,X_2,\\dots,X_n)$ 是样本，于是似然函数\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "L(\\lambda) &= L(x_1,x_2,\\dots,x_n;\\lambda)\\\\\n",
    "&=\\Pi_{i=1}^{n}(\\frac{\\lambda^{x_i}e^{-\\lambda}}{{x_i}!})\\\\\n",
    "&={e^{-n\\lambda}\\lambda^{\\sum_{i=1}^{n}x_i}\\over \\Pi_{i=1}^{n}x_i!}\n",
    "\\end{align}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "两边取对数得\n",
    "\n",
    "$$\n",
    "\\ln L(\\lambda) =-n\\lambda+\\ln \\lambda\\sum_{i=1}^{n}x_i-\\sum_{i=1}^{n}\\ln(x_i!)\n",
    "$$\n",
    "\n",
    "令\n",
    "\n",
    "$$\n",
    "\\frac{d\\ln L(\\lambda)}{d\\lambda} = 0\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "解得\n",
    "\n",
    "$$\n",
    "\\widehat \\lambda = \\overline x\\\\\n",
    "\\frac{d^2\\ln L(\\lambda)}{d\\lambda^2}|_{\\widehat \\lambda = \\overline x} < 0\n",
    "$$\n",
    "\n",
    "从而得出 $\\lambda$ 的极大似然估计为\n",
    "\n",
    "$$\n",
    "\\widehat \\lambda = \\overline x\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例14** 设总体 $X$ 服从参数为 $\\lambda$ 的指数分布, 其中 $\\lambda$ 未知, $(x_1,x_2,\\dots,x_n)$ 为其样本, 试求参数 $\\lambda$ 的极大似然估计值和估计量."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:** 总体 $X$ 服从参数为 $\\lambda$ 的指数分布, 则有\n",
    "\n",
    "$$\n",
    "f(\\lambda,x) =\n",
    "\\begin{cases}\n",
    "\\lambda e^{-\\lambda x},&x>0\\\\\n",
    "0,&x\\leq 0\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "所以似然函数为\n",
    "\n",
    "$$\n",
    "L(\\lambda) = \\lambda^ne^{-\\lambda\\sum_{i=1}^{n}x_i}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "取对数\n",
    "\n",
    "$$\n",
    "\\ln L(\\lambda) =n\\ln \\lambda-\\lambda\\sum_{i=1}^{n}x_i\n",
    "$$\n",
    "\n",
    "求导并令其等于 $0$, 解得 $\\lambda$ 的极大似然估计值为\n",
    "\n",
    "$$\n",
    "\\widehat \\lambda = \\frac{n}{\\sum_{i=1}^{n}x_i}=\\frac{1}{\\overline x}\n",
    "$$\n",
    "\n",
    "极大似然估计量为\n",
    "\n",
    "$$\n",
    "\\widehat \\lambda = \\frac{n}{\\sum_{i=1}^{n}x_i}=\\frac{1}{\\overline x}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例15** 设 $(x_1,x_2,\\dots,x_n)$ 是来自正态总体 $N(\\mu,\\sigma^2)$ 的一个样本,其中 $\\mu,\\sigma^2$ 是未知参数,参数空间\n",
    "\n",
    "$$\n",
    "\\theta = \\{-\\infty<\\mu<\\infty,\\sigma^2>0\\}\n",
    "$$\n",
    "\n",
    "求 $\\mu,\\sigma^2$ 的极大似然估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:** 正态分布的似然函数为\n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "L&=L(\\mu,\\sigma^2;x_1,x_2,\\dots,x_n)\\\\\n",
    "&=(\\frac{1}{2\\pi\\sigma^2})^{n\\over 2} e^{-\\frac{1}{2\\sigma^2}\\sum_{i=1}^{n}(x_i-\\mu)^2}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    "两边取对数得\n",
    "\n",
    "$$\n",
    "\\ln L =-\\frac{n}{2}\\ln(2\\pi)-\\frac{n}{2}\\ln \\sigma^2-\\frac{1}{2\\sigma^2}\\sum_{i=1}^{n}(x_i-\\mu)^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "分别关于 $\\mu$ 和 $\\sigma^2$ 求偏导数，得似然方程组\n",
    "\n",
    "$$\n",
    "\\begin{cases}\n",
    "\\frac{\\partial \\ln L}{\\partial\\mu}=\\frac{1}{\\sigma^2}\\sum_{i=1}^{n}(x_i-\\mu)=0\\\\\n",
    "\\frac{\\partial \\ln L}{\\partial\\sigma^2}=-\\frac{n}{2\\sigma^2}+\\frac{1}{2\\sigma^4}\\sum_{i=1}^{n}(x_i-\\mu)^2=0\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "解得\n",
    "\n",
    "$$\n",
    "\\widehat \\mu = \\overline x\\\\\n",
    "\\widehat \\sigma^2 = \\frac{1}{n}\\sum_{i=1}^{n}(x_i-\\mu)^2\n",
    "$$\n",
    "\n",
    "易验证以上所求为 $\\mu$ 和 $\\sigma^2$ 的极大似然估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例16**  设总体 $X$ 具有均匀分布, 其概率密度函数为\n",
    "\n",
    "$$\n",
    "f(x;\\theta)=\n",
    "\\begin{cases}\n",
    "\\frac{1}{\\theta},&0<x\\leq\\theta\\\\\n",
    "0,&其他\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "其中 $0<\\theta<\\infty$, 求未知参数 $\\theta$ 的极大似然估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:** 设 $(X_1,\\dots,X_n)$ 是总体 $X$ 的样本，似然函数为\n",
    "\n",
    "$$\n",
    "L(\\theta;x_1,x_2,\\dots,x_n) = \\frac{1}{\\theta^n}, 0<x_i<\\theta,i=1,2,\\dots,n\n",
    "$$\n",
    "\n",
    "要使 $L(\\theta;x_1,x_2,\\dots,x_n)$ 达到最大，只要 $\\theta$ 最小，由于"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\n",
    "0<x_i<x_{(n)}=\\max_{1\\leq i\\leq n}\\{x_i\\}\\leq\\theta,i=1,2,\\dots,n\n",
    "$$\n",
    "\n",
    "所以 $\\theta$ 的极大似然估计值为:\n",
    "\n",
    "$$\n",
    "\\widehat \\theta(X_1,\\dots,X_n)=\\max_{1\\leq i\\leq n}\\{X_i\\}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**例17** 设 $x_1,x_2,\\dots,x_n$ 是取自总体 $X$ 的一个样本\n",
    " \n",
    "$$\n",
    "X～f(x;\\theta)=\n",
    "\\begin{cases}\n",
    "\\frac{1}{\\theta}e^{-(x-\\mu)/\\theta},&x\\geq\\mu\\\\\n",
    "0,&其他\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "其中 $\\theta$, $\\mu$ 为未知参数， $\\theta>0$,求 $\\theta$, $\\mu$ 的极大似然估计."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**解:**似然函数为\n",
    "\n",
    "$$\n",
    "L(\\theta,\\mu) = \n",
    "\\begin{cases}\n",
    "\\Pi_{i=1}^{n}\\frac{1}{\\theta}e^{-(x_i-\\mu)/\\theta},&x_i\\geq\\mu,i=1,2,\\dots,n\\\\\n",
    "0,&\\text{ 其他}\n",
    "\\end{cases}\\\\\n",
    "=\\begin{cases}\n",
    "\\frac{1}{\\theta^n}e^{-\\frac{1}{\\theta}\\sum_{i=1}^{n}(x_i-\\mu)},&\\min x_i\\geq\\mu\\\\\n",
    "0,&\\text{其他}\n",
    "\\end{cases}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "对数似然函数为\n",
    "\n",
    "$$\n",
    "\\ln L(\\theta,\\mu) = -n\\ln\\theta-\\frac{1}{\\theta}\\sum_{i=1}^{n}(x_i-\\mu)\n",
    "$$\n",
    "\n",
    "对 $\\theta,\\mu$ 分别求导并令其 $0$, 得"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "$$\n",
    "\\frac{\\partial \\ln L(\\theta,\\mu)}{\\partial\\theta}=0\\qquad(1)\\\\\n",
    "\\frac{\\partial \\ln L(\\theta,\\mu)}{\\partial\\mu}=0\\qquad(2)\n",
    "$$\n",
    "\n",
    "由 (1) 得\n",
    "\n",
    "$$\n",
    "\\theta = \\frac{1}{n}\\sum_{i=1}^{n}x_i\n",
    "-\\mu\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "由于\n",
    "\n",
    "$$\n",
    "L(\\theta,\\mu) =\n",
    "\\begin{cases}\n",
    "\\frac{1}{\\theta^n}e^{-\\frac{1}{\\theta}\\sum_{i=1}^{n}(x_i-\\mu)},&\\min x_i\\geq\\mu\\\\\n",
    "0,&\\text{其他}\n",
    "\\end{cases}\n",
    "$$\n",
    "\n",
    "对 $\\mu\\leq \\min x_i$, $L(\\theta,\\mu)>0$, 且是 $\\mu$ 的增函数; 当 $\\mu$ 取其他值时, $L(\\theta,\\mu)= 0$.\n",
    "\n",
    "故使 $L(\\theta,\\mu)$ 达到最大的 $\\mu$, 即 $\\mu$ 的 MLE 是\n",
    "\n",
    "$$\n",
    "\\mu^*=\\min_{1\\leq i\\leq n}x_i\n",
    "$$\n",
    "\n",
    "即 $\\theta^*$, $\\mu^*$ 为 $\\theta$, $\\mu$ 的MLE. 于是 $\\theta^* = \\frac{1}{n}\\sum_{i=1}^{n}x_i-\\mu^*$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "这一节, 我们介绍了参数点估计, 讨论了估计量的优良性准则. 给出了寻求估计量最常用的矩法和极大似然法. 参数点估计是用一个确定的值去估计未知的参数. 看来似乎精确, 实际上把握不大.为了使估计的结论更可信, 需要引入区间估计. 这是下一节的内容."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "**小结**\n",
    "\n",
    "求未知参数估计量的常用方法是矩估计法和极大似然估计法.\n",
    "\n",
    "**矩估计法:**以样本矩作为总体的相应矩的估计,以样本矩的函数作为总体的相应矩的函数的估计.\n",
    "\n",
    "**极大似然法:**使似然函数达到最大值.似然函数对离散情形考虑概率,连续情形考虑概率密度."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  },
  "livereveal": {
   "height": 768,
   "scroll": "True",
   "slideNumber": "True",
   "start_slideshow_at": "selected",
   "theme": "sky",
   "width": 1024
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}