lecture_4_calculus_2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create anaconda environment\n",
    "<br>\n",
    "```bash\n",
    "conda create -n ml python=3.7.4 jupyter\n",
    "```\n",
    "Install fastai library\n",
    "<br>\n",
    "```bash\n",
    "conda install -c pytorch -c fastai fastai\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Derivatives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $U \\subset \\mathbb{R}$ be a open subset and $f:U \\subset \\mathbb{R} \\to \\mathbb{R} $:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function has a derivative at $x_0 \\in \\mathbb{R}$ if for every $h \\in \\mathbb{R}$ there exists the limit of $$\\lim_{h \\to 0}\\frac{f(x_0 + h) - f(x_0)}{h}$$\n",
    "<br>\n",
    "This limit point is called derivate at point $x_0$ and denoted by $f'(x_0)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/calc2/derivat1.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/calc2/derivat2.gif \"segment\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the property of limits we can write: \n",
    "$$\\lim_{h\\to 0}\\frac{f(a+h) - f(a))}{h} = f'(a)$$ \n",
    "then \n",
    "$$\\lim_{h\\to 0}\\frac{f(a+h) - f(a))}{h} - f'(a) = 0$$\n",
    "$$\\lim_{h\\to 0}\\frac{f(a+h) - f(a))}{h} - \\lim_{h\\to 0} f'(a) = 0$$\n",
    "which implies: $$\\lim_{h\\to 0}\\frac{f(a+h) - f(a) - h \\cdot f'(a)}{h} = 0$$\n",
    "and \n",
    "$$\n",
    "\\lim_{h\\to 0}\\frac{f(a+h) - (f(a) + f'(a)\\cdot h)}{h} = 0\n",
    "$$\n",
    "We can consider that:\n",
    "$$\n",
    "f(a+h) \\approx f(a) + f'(a)h\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If function is differentiable at each point of $U$ then we can define derivative function $f':U \\subset \\mathbb{R} \\to \\mathbb{R}$ as $f':x \\mapsto f'(x)$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: $f(x) = x^a$ then $f'(x) = ax^{a-1}$ for any $a \\in \\mathbb{R}$\n",
    "<br>\n",
    "Example: If $f(x) = a$ is a constant function then $f'(x) = 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Properties of derivatives:\n",
    "- $(f + g)'(x) = f'(x) + g'(x)$\n",
    "- $(af(x))' = af'(x)$\n",
    "- $(fg)'(x) = f'(x)g(x) + f(x)g'(x)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The chain rule: Let $f:\\mathbb{R} \\to \\mathbb{R}$ and $g:\\mathbb{R} \\to \\mathbb{R}$ then for $g \\circ f:\\mathbb{R} \\to \\mathbb{R}$ derivative $(g \\circ f)'(x) = g'(f(x)) f'(x)$ and for derivative function we have: $(g \\circ f)' = g'_{f}f'$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: $(g \\circ f)(x) = (ax - b)^2$ then we have $f(x) = ax - b$ and $g(y) = y^2$ and for composition, we can consider $ax - b$ as a single variable and we get $2(ax - b)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Partial derivatives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let now $f:\\mathbb{R}^n \\to \\mathbb{R}^m$ be a function between two normed vector spaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The partial derivative of a function $f$ ($x_1, \\dots, x_i$) in the direction $x_i$ at the point ($a_1, \\dots, a_n$) is defined to be:\n",
    "\n",
    "$$\n",
    "\\frac{\\partial f}{\\partial x_i}(a_1, \\ldots, a_n) = \\lim_{h \\to 0}\\frac{f(a_1, \\ldots, a_i+h,\\ldots,a_n) - f(a_1,\\ldots, a_i, \\dots,a_n)}{h}\n",
    "$$\n",
    "\n",
    "All the variables are fixed except $x_i$. That choice of fixed values determines a function of one variable\n",
    "\n",
    "$$f_{a_1,\\ldots,a_{i-1},a_{i+1},\\ldots,a_n}(x_i) = f(a_1,\\ldots,a_{i-1},x_i,a_{i+1},\\ldots,a_n)$$\n",
    "\n",
    "and by definition,\n",
    "\n",
    "$$\\frac{df_{a_1,\\ldots,a_{i-1},a_{i+1},\\ldots,a_n}}{dx_i}(a_i) = \\frac{\\partial f}{\\partial x_i}(a_1,\\ldots,a_n)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider a vector $f(x) \\in \\mathbb{R}^m$ for some $x\\in \\mathbb{R}^n$ and function $f:\\mathbb{R}^n \\to \\mathbb{R}^m$, then $f(x) = (f_{1}(x), f_{2}(x), \\dots, f_{m}(x)) $ then partial derivative can be considered as:\n",
    "<br>\n",
    "$$\\frac{\\partial f}{\\partial x_i}(a_1, \\ldots, a_n) = \\frac{\\partial f}{\\partial x_i}(a) = \\lim_{h \\to 0}\\frac{1}{h}(f(a_1, \\ldots, a_i+h,\\ldots,a_n) - f(a_1,\\ldots, a_i, \\dots,a_n))$$\n",
    "<br>\n",
    "So following the rules of vectors:\n",
    "$$\\frac{\\partial f}{\\partial x_i}(a_1, \\ldots, a_n) = (\\lim_{h \\to 0}\\frac{f_{1}(a_1, \\ldots, a_i+h,\\ldots,a_n) - f_{1}(a_1,\\ldots, a_i, \\dots,a_n)}{h}, \\lim_{h \\to 0}\\frac{f_{2}(a_1, \\ldots, a_i+h,\\ldots,a_n) - f_{2}(a_1,\\ldots, a_i, \\dots,a_n)}{h}, \\dots, \\lim_{h \\to 0}\\frac{f_{m}(a_1, \\ldots, a_i+h,\\ldots,a_n) - f_{m}(a_1,\\ldots, a_i, \\dots,a_n)}{h})\n",
    "$$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define partial derivatives $$\\mathbf{J} = \\begin{pmatrix}\n",
    "    \\dfrac{\\partial \\mathbf{f}}{\\partial x_1} & \\cdots & \\dfrac{\\partial \\mathbf{f}}{\\partial x_n} \\end{pmatrix}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So for each coordinate of functions values we then get matrix:\n",
    "<br>\n",
    "$$\\mathbf{J} = \n",
    "\\begin{pmatrix}\n",
    "    \\dfrac{\\partial f_1}{\\partial x_1} & \\cdots & \\dfrac{\\partial f_1}{\\partial x_n}\\\\\n",
    "    \\vdots & \\ddots & \\vdots\\\\\n",
    "    \\dfrac{\\partial f_m}{\\partial x_1} & \\cdots & \\dfrac{\\partial f_m}{\\partial x_n}\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: \n",
    "$$f(x_1, x_2, x_3) = (2x_{1}^{2} + x_2, 0.5x_{3} + 4)$$ then we have a \n",
    "$$f : \\mathbb{R}^{?} \\to \\mathbb{R}^{?}$$\n",
    "and\n",
    "$$\\mathbf{J} = \n",
    "\\begin{pmatrix}\n",
    "    ?\\\\\n",
    "    \\vdots & \\ddots & \\vdots\\\\\n",
    "    ?\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Example: \n",
    "$$f(x_1, x_2, x_3) = (2x_{1}^{2} + x_2, 0.5x_{3} + 4)$$ \n",
    "and\n",
    "$$g(x_1, x_2) = (x_{1}^{4}, 10x_{1}^{3} + 8x_{2}^{2})$$\n",
    "then we have a \n",
    "$$(g \\circ f) : \\mathbb{R}^{?} \\to \\mathbb{R}^{?}$$\n",
    "and\n",
    "$$\\mathbf{J} = \n",
    "\\begin{pmatrix}\n",
    "    ?\\\\\n",
    "    \\vdots & \\ddots & \\vdots\\\\\n",
    "    ?\n",
    "\\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Jacobian of composition:\n",
    "$f : \\mathbb{R}^n \\to \\mathbb{R}^n$ and $g : \\mathbb{R}^m \\to \\mathbb{R}^k$ then chain rule generalizes for Jacobian matrices, \n",
    "$$\\mathbf{J}_{\\mathbf{g} \\circ \\mathbf{f}}(\\mathbf{x}) = \\mathbf{J}_{\\mathbf{g}}(\\mathbf{f}(\\mathbf{x})) \\mathbf{J}_{\\mathbf{f}}(\\mathbf{x})\n",
    "$$\n",
    "for $x \\in \\mathbb{R}^n$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Directional derivative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let for open subset $U$ of $\\mathbb{R}^n$ and function $f: U \\subset \\mathbb{R}^n \\to \\mathbb{R}$  ($f(\\mathbf{x}) = f(x_1, x_2, \\ldots, x_n)$) and for any vector $\\mathbf{v} \\in \\mathbb{R}^n$ define:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\nabla_{\\mathbf{v}}{f}(\\mathbf{x}) = \\lim_{h \\rightarrow 0}{\\frac{f(\\mathbf{x} + h\\mathbf{v}) - f(\\mathbf{x})}{h}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Total derivative"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define generalization derivative of function $f:U \\subset \\mathbb{R}^n \\to \\mathbb{R}^m$ defined of open $U$ in manner:\n",
    "The function $f$ is differentiable in $x \\in U$ if there exists\n",
    "$$df_a:\\mathbb{R}^n \\rightarrow \\mathbb{R}^m$$ liner transformation such that:\n",
    "$$\n",
    "\\lim_{x\\rightarrow a}\\frac{\\|f(x)-f(a)-df_a(x-a)\\|}{\\|x-a\\|}=0\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As in the case of derivatives, we can consider that:\n",
    "$$f(a + h) \\approx f(a) + df_a(h)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Property: The function is differentiable if and only if each $f_i \\colon U \\to \\mathbf{R}$ is differentiable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Property: If function is differentiable then it has all partial derivatives $\\partial f/\\partial x_i$ for each $i \\in \\{1, \\dots , n\\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If $\\partial f/\\partial x_i$ for each $i \\in \\{1, \\dots , n\\}$ and are continuous in some neighbourhood of $x_i$ then there exists total derivative and it corresponds to the Jacobian matrix."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example:\n",
    "$$f(x,y) = \\begin{cases}x & \\text{if }y \\ne x^2 \\\\ 0 & \\text{if }y = x^2\\end{cases}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Example:\n",
    "$$f(x,y) = \\begin{cases}y^3/(x^2+y^2) & \\text{if }(x,y) \\ne (0,0) \\\\ 0 & \\text{if }(x,y) = (0,0)\\end{cases}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient of the function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Considre the function $f:\\mathbb{R}^n \\to \\mathbb{R}$ and consider Jacobian matrix for partial derivatives:\n",
    "$$\n",
    "\\mathbf{J} = \\begin{pmatrix} \\frac{\\partial f}{\\partial x_1}, & \\cdots & , & \\frac{\\partial f}{\\partial x_n} \\end{pmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We call this vector the gradient of the function and denote it by $\\nabla{f}$ or by $(\\nabla f)_{x_0}$ for gradient in point $x_0 \\in \\mathbb{R}^n$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient in $x_0$ can be considered as linear approximation of the function in point $x_0$\n",
    "$$f(x) \\approx f(x_0) + (\\nabla f)_{x_0}\\cdot(x-x_0)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient and directional derivative:\n",
    "$$\\big(\\nabla f(x)\\big)\\cdot \\mathbf{v} = D_{\\mathbf v}f(x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient and differential - if function is differentiable in x\n",
    "$$(\\nabla f)_x\\cdot v = df_x(v)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Properties:\n",
    "- $\\nabla\\left(\\alpha f+\\beta g\\right)(a) = \\alpha \\nabla f(a) + \\beta\\nabla g (a)$\n",
    "- $\\nabla (fg)(a) = f(a)\\nabla g(a) + g(a)\\nabla f(a)$\n",
    "Chain rule:\n",
    "- $(f\\circ g)'(c) = \\nabla f(a)\\cdot g'(c)$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Function surface and gradient vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/multivariate_function_gradient.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient will point to the steepest slope and its magnitude gives is the rate of increase in that direction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\big(\\nabla f(x)\\big)\\cdot \\mathbf{v} = D_{\\mathbf v}f(x)$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local extremums"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We say that point $x_0 \\in U$ is a local minima of the function $f:U \\subset \\mathbb{R}^n \\to \\mathbb{R}$ if there exists such $r \\in \\mathbb{R}_+$ such that $f(x) \\ge f(x_0)$\n",
    "for each $x \\in \\mathbf{B}(x_0, r)$ or for each $∣∣x−x_0∣∣<r$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "In the same manner, we can define local maxima"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let for $f:U \\subset \\mathbb{R}^n \\to \\mathbb{R}$ fix all variables except $x_i$ and consider function $f(x_i)|_{x_1, x_2, \\dots, x_{i-1}, x_{i+1}, \\dots x_i}: U_i \\subset \\mathbb{R} \\to \\mathbb{R}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The point $x_0$ is local extrema if and only if it is for each projection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First is easy, for the second we can take the $r = min \\{r_i | i \\in (1, 2, \\dots, n)\\}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local minima example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/extreme_points_local_minimas.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local maxima example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/extreme_points_local_maximas.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient directions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gradient always points to the steepest point of the function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we consider projection of the gradient on arbitrary unit vector $\\nabla{f} \\cdot u = ||\\nabla{f}|| \\cdot ||u|| \\cdot \\cos{\\alpha}$ for unit vector we have that $||u|| = 1$ and thus $\\nabla{f} \\cdot u = ||\\nabla{f}|| \\cdot \\cos{\\alpha}$ and this is maximal when $\\cos{\\alpha} = 1$ or when unit and gradient vectors have the same directions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fermat's theorem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If function $f\\colon (a,b) \\rightarrow \\mathbb{R}$ has a local extrema at some point $x_0 \\in (a,b)$ and $f$ is differentiable at $x_0$ then $f'(x_0) = 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For gradients, we have the same if $f\\colon U \\subset \\mathbb{R}^n \\to \\mathbb{R}$ and has a local extrema at some point $x_0 \\in U$ and $f$ is differentiable at $x_0$ then $\\nabla{f}(x_0) = 0$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saddle points"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/saddle_point.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this points gradient is also zero"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The question is how can we find local minima point of a function of hundreds of million parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first calculate the gradient of this function, we know that it points to the local maxima so we can take inverce of the gradient and step by step change variables in this direction "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to speed up our descent, we can multiply the inverse of the gradient on some positive real number $\\alpha \\in \\mathbb{R}_+$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In one dimensional case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\Delta x_0 =x_0 -\\alpha \\cdot f'(x_0)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/single_variable_function_gradient_directions.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we calculate $\\Delta X =X_0 -\\alpha \\cdot \\nabla{f}(X_0)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In multidimensional case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/multivariate_function_gradient_directions.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient descent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/calc2/gradient_descent_first_order.gif \"gradient descent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Gradient descent speed depends on \"smoothness\" of the surface of function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/calc2/gradient_descent_first_order_2.gif \"gradient descent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For\n",
    "$$f: U \\subset \\mathbb{R}^n \\to \\mathbb{R}$$\n",
    "Define\n",
    "$$\\nabla{f}: U \\subset \\mathbb{R}^n \\to \\mathbb{R}^n$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/vectorf1.jpg\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saddle point"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/saddle_point.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hessian matrix of continuous functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\mathbf H = \\begin{bmatrix}\n",
    "  \\dfrac{\\partial^2 f}{\\partial x_1^2} & \\dfrac{\\partial^2 f}{\\partial x_1\\,\\partial x_2} & \\cdots & \\dfrac{\\partial^2 f}{\\partial x_1\\,\\partial x_n} \\\\[2.2ex]\n",
    "  \\dfrac{\\partial^2 f}{\\partial x_2\\,\\partial x_1} & \\dfrac{\\partial^2 f}{\\partial x_2^2} & \\cdots & \\dfrac{\\partial^2 f}{\\partial x_2\\,\\partial x_n} \\\\[2.2ex]\n",
    "  \\vdots & \\vdots & \\ddots & \\vdots \\\\[2.2ex]\n",
    "  \\dfrac{\\partial^2 f}{\\partial x_n\\,\\partial x_1} & \\dfrac{\\partial^2 f}{\\partial x_n\\,\\partial x_2} & \\cdots & \\dfrac{\\partial^2 f}{\\partial x_n^2}\n",
    "\\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis of Hessian matrix, Schwartz’s theorem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tailor’s series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$f(a)+\\frac {f'(a)}{1!} (x-a)+ \\frac{f''(a)}{2!} (x-a)^2+\\frac{f'''(a)}{3!}(x-a)^3+ \\cdots$$\n",
    "\n",
    "$$\\sum_{n=0} ^ {\\infty} \\frac {f^{(n)}(a)}{n!} (x-a)^{n}.$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/taylor_series_sin_approximation.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Newton’s method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/calc2/newton_method.gif \"gradient descent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/calc2/newton_method_2.gif \"gradient descent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reverse mode differentiation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance $f:U\\subset \\mathbb{R}^n \\to \\mathbb{R}$\n",
    "<br>\n",
    "$$f = f_{L} \\circ f_{L-1} \\circ \\dots \\circ f_{1}$$\n",
    "where\n",
    "$$f_{i}(X) = \\sigma(W^{i}X^{T} + b)$$\n",
    "$W^{i} \\in \\mathbb{R}^{n_i \\times n_{i+1}}$, $X^{i} \\in \\mathbb{R}^{1  \\times n^{i+1}}$ and $b \\in \\mathbb{R}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we want to calculate the gradient with respect to $$W = (W^{1}, W^{2}, \\dots , W^{L})$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reverse Mode Differentiation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial y}{\\partial x}\n",
    "= \\frac{\\partial y}{\\partial w_{n-1}} \\frac{\\partial w_{n-1}}{\\partial x}\n",
    "= \\frac{\\partial y}{\\partial w_{n-1}} \\left(\\frac{\\partial w_{n-1}}{\\partial w_{n-2}} \\frac{\\partial w_{n-2}}{\\partial x}\\right)\n",
    "= \\frac{\\partial y}{\\partial w_{n-1}} \\left(\\frac{\\partial w_{n-1}}{\\partial w_{n-2}} \\left(\\frac{\\partial w_{n-2}}{\\partial w_{n-3}} \\frac{\\partial w_{n-3}}{\\partial x}\\right)\\right)\n",
    "= \\cdots\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/reverse_mode_differentiation.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "differentiation in reverse order and cache the results once"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/reverse1.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then for instant values we have"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/reverse2.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\n",
    "\\frac{\\partial}{\\partial a}(a+b) = \\frac{\\partial a}{\\partial a} + \\frac{\\partial b}{\\partial a} = 1\n",
    "$$\n",
    "<br>\n",
    "$$\n",
    "\\frac{\\partial}{\\partial u}uv = u\\frac{\\partial v}{\\partial u} + v\\frac{\\partial u}{\\partial u} = v\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/calc2/reverse3.png\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}