S02a_Descriptive_Statistics.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Execute before using this notebook if using google colab\n",
    "\n",
    "kernel = str(get_ipython())\n",
    "\n",
    "if 'google.colab' in kernel:    \n",
    "    !wget https://raw.githubusercontent.com/fredzett/rmqa/master/utils.py -P local_modules -nc \n",
    "    !npx degit fredzett/rmqa/data data\n",
    "    import sys\n",
    "    sys.path.append('local_modules')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning goals\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is a refresher on basic **descriptive statistics**. It is assumed that this material has been covered in an undergraduate statistics course before. We will use our existing knowledge and practically apply it in Python. \n",
    "\n",
    "To this end, the main learning goals of this Chapter are:\n",
    "\n",
    "**A/ Provide you with a quick refresher on**\n",
    "\n",
    "- measures of location: mean, median, percentiles, quartiles\n",
    "- measures of variability: range, variance, standard deviation\n",
    "\n",
    "**B/ Practice coding in Python**\n",
    "\n",
    "- we will implement most concepts from scratch\n",
    "- we will learn that most concepts applied in this course have already been programmed in python and packaged into modules\n",
    "- we will start using some relevant modules\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measures of location"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A location parameter of a (probability) distribution is a scalar parameter, which determines the \"location\" or shift of the distribution. \n",
    "\n",
    "We will be using the following monthly sales data from **Statistics Inc.**:\n",
    "\n",
    "| Month | Sales |\n",
    "|:-----:|:-----:|\n",
    "|   1   |  2020 |\n",
    "|   2   |  2075 |\n",
    "|   3   |  2125 |\n",
    "|   4   |  2040 |\n",
    "|   5   |  1980 |\n",
    "|   6   |  1955 |\n",
    "|   7   |  2050 |\n",
    "|   8   |  2165 |\n",
    "|   9   |  2070 |\n",
    "|   10  |  2260 |\n",
    "|   11  |  2060 |\n",
    "|   12  |  2040 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_sales(): \n",
    "    return [2020,2075, 2125, 2040, 1980, 1955, 2050, 2165, 2070, 2260, 2060, 2040]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales = load_sales()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Likely the most important and most known measure of location is the `mean`, also called average value for a given variable.\n",
    "\n",
    "Two different means depending on the data:\n",
    "\n",
    "1. if data from a sample: mean is denoted by $\\bar{x}$\n",
    "\n",
    "2. if data from a population: mean is denoted by $\\mu$\n",
    "\n",
    "The value of variable $X$ for the $i^{th}$ observation is denoted by $x_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Sample mean** for a sample with $n$ observations:\n",
    "\n",
    "$$\\bar{x} = \\frac{\\sum{x_i}}{n}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Population mean** is the same (except for different notation):\n",
    "\n",
    "$$\\mu = \\frac{\\sum{x_i}}{N}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "where \n",
    "- sample: a set that contains some members of a population \n",
    "- population: a set that contains all members of a group\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> **Note** The result for sample and population mean is identical. We use different notation only to indicate whether mean is from population or sample. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mean can easily be calculated using the below python code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2070.0"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_bar = sum(sales)/len(sales)\n",
    "x_bar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we intend to apply the calculation of a mean on various variables, it is handy to put the concept into a function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean(lst):\n",
    "    return sum(lst)/len(lst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2070.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mean(sales)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Median"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The median is another measure of location. The median is the middle value when data is sorted in ascending order. \n",
    "\n",
    "The **median** of a variable is defined as:\n",
    "  \\begin{equation}\n",
    "    median=\n",
    "    \\begin{cases}\n",
    "      x_{((n + 1)/2)}, & \\text{if $n$ is odd} \\\\\n",
    "      \\dfrac{1}{2}\\big(x_{n/2} + x_{(n/2 + 1)}\\big), & \\text{if $n$ is even}\n",
    "    \\end{cases}\n",
    "  \\end{equation}\n",
    "\n",
    "The median is less sensitive to outliers than the **mean** and hence is a more robust measure of the center of a distribution.\n",
    "\n",
    "Therefore, the following steps need to be processed to calculate the median\n",
    "\n",
    "1. Arrange the data in ascending order\n",
    "2. Determine number of elements\n",
    "3. Calculate median:\n",
    "    1. for an odd number of observations, the median is the middle value\n",
    "    2. for an even number of observations, the median is the average of the two middle values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Naive implementation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A simple and naive implementation of mean would look as follows:\n",
    "\n",
    "Elements: 0 1 2 3 4 **5 6** 7 8 9 10 11\n",
    "\n",
    "We need to take the average of indices 5 and 6. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "(sales[5] + sales[6])/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean([sales[5], sales[6]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This approach is, however, not reproducabel with other lists. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### **Better implementation**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true,
    "toc-nb-collapsed": true
   },
   "source": [
    "Let's calculate the mean of the sales data by following the above steps in a more general form. We need to introduce \n",
    "\n",
    "- `list.sort()`: a method (i.e. function) for lists sorting lists in ascending order\n",
    "- `comparison operators`: operators to compare two values (e.g. if a is greater than b)\n",
    "- `if conditions`: instruct python what to do something if a statement is true or false"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. Sort sales data in ascending order**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales.sort()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Determine number of elements**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n = len(sales)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Calculate median depending by checking if odd or even number of elements**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if n % 2 == 0: # List has even number of elements (used remainder operator to check this)\n",
    "    el1 = sales[n//2] # Floor division 7 / 2 = 3.5; 7 // 2 = 3\n",
    "    el2 = sales[n//2 - 1]\n",
    "    median = (el1 + el2)/2\n",
    "else: # List has odd number of elements\n",
    "    median = sales[n//2]\n",
    "median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def median(lst):\n",
    "    lst.sort()\n",
    "    n = len(lst)\n",
    "    \n",
    "    if n % 2 == 0: # List has even number of elements\n",
    "    \n",
    "        el1 = lst[n//2]\n",
    "        el2 = lst[n//2 - 1]\n",
    "        median = (el1 + el2)/2\n",
    "    \n",
    "    else: # List has odd number of elements\n",
    "        \n",
    "        median = lst[n//2]\n",
    "    \n",
    "    return median"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers = [1,2,3,4,5,6]\n",
    "median(numbers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc-hr-collapsed": true,
    "toc-nb-collapsed": true
   },
   "source": [
    "Advantage of median of mean: mean is influenced by small or large outliers. This is not the case for the median. \n",
    "\n",
    "Example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_with_outliers = sales + [100_000]\n",
    "sales_with_outliers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mean(sales_with_outliers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "median(sales_with_outliers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another measure of location is the `mode`. \n",
    "\n",
    "**Mode**: the mode is the value that occurs most often within a set of numbers.\n",
    "\n",
    "If there are no repeats in the data the mode is not defines. \n",
    "\n",
    "The mode is an important measure of location for qualitative data (i.e. non numeric data). A data set can have more than one mode. A data set with two modes is called bimodal. A data set with more than two modes is called multimodal. \n",
    "\n",
    "Similar to the median the mode is not affected by extreme outlieres."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation from scratch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While the statistical concept is straight forward the implementation in Python is not trivial for beginners.\n",
    "\n",
    "Here is one possible implementation from scratch (don't worry if it is not entirely clear to you)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max(set(sales), key=sales.count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation using modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have reached a point where it becomes quite tedious to implement all concept from scratch. Luckily, most of the concepts we will be using in this class (or actually most concepts in statistics or data analysis) have already been implemented in Python and can `imported` into python. \n",
    "\n",
    "We need to introduce a new concept:\n",
    "\n",
    "- `import modules`: use (e.g. statistical) concepts that have already been implemented in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "statistics.mode(sales)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "letters = [\"a\",\"b\",\"b\",\"a\",\"d\",\"a\",\"a\",\"c\"]\n",
    "statistics.mode(letters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Percentiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A percentile provides information about how the data is spread over the interval from smallest value to largest value.\n",
    "\n",
    "> **Percentile** is defined as:\n",
    "> \n",
    "> The $p^{th}$ percentile is a value such that *at least* $p$ percent of the observations are less than or equal to this value and *at least* $(100 - p)$ per cent of the observations are greater than the value.\n",
    "\n",
    "In statistics percentiles are often used to \n",
    "\n",
    "(i) express the spread of data and \n",
    "\n",
    "(ii) to remove outliers (e.g. every value $> 95^{th}$ percentile is removed)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Calculation** of the $p^{th}$ percentile is done in three steps:\n",
    "\n",
    "1. Sort the data in ascending order (i.e. smallest to largest value)\n",
    "\n",
    "2. Compute an index $i$\n",
    "\n",
    "$$i = \\bigg(\\frac{p}{100}\\bigg)\\times n$$\n",
    "\n",
    "where $p$ is the percentiles of interest and n is the number of observations. \n",
    "\n",
    "3. Two options:\n",
    "\n",
    "    - $i$ *is not* an integer: the next integer greater than $i$ denotes the position of the $p^{th}$ percentile\n",
    "\n",
    "    - $i$ *is* an interger: the $p^{th}$ percentile is the average of the values in positions $i$ and $i + 1$\n",
    "\n",
    "\n",
    "\n",
    "**Note** \n",
    "\n",
    "- there are different conventions to calculate percentiles (e.g. round down if $i$ is not an integer). The specific method become less important the more observations we have. \n",
    "\n",
    "- the 50th percentile is equal to the median\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use our sales data and calculate the $85^{th}$ percentile. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales = load_sales() # Reload clean sales data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 1:** Sort the data in ascending order (using `sort`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales.sort()\n",
    "sales"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 2:** compute index $i$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "idx = (85/100)*len(sales)\n",
    "idx = idx - 1 # Note: in Python the first position has the index 0 (not 1)\n",
    "idx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Step 3:** calculate percentile\n",
    "\n",
    "First, we need to determine if `idx` is an integer. Second we need to round up if the number is not an integer. For this we need one new function and one new module\n",
    "\n",
    "- `is_integer()`: To check if a number is an integer (i.e. a whole number)\n",
    "- `import math`: import module with math functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math \n",
    "\n",
    "if idx.is_integer() == False: # Note: use stackoverflow to get help\n",
    "    position = math.ceil(idx)\n",
    "    percentile = sales[position]\n",
    "else:\n",
    "    idx = int(idx)\n",
    "    percentile = (sales[idx] + sales[idx +1 ])/2\n",
    "percentile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$85\\%$ of all sales values are below $2.165$. \n",
    "\n",
    "Let's put everything together into one function. The function should take two arguments:\n",
    "\n",
    "- list of values: denoted `lst`\n",
    "- $p^{th}$ percentile: denoted `p`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def percentile(lst, p):\n",
    "    lst.sort()\n",
    "    idx = (p/100)*len(lst) - 1\n",
    "    \n",
    "    if idx.is_integer() == False:\n",
    "        position = math.ceil(idx)\n",
    "        percentile = lst[position]\n",
    "    else:\n",
    "        idx = int(idx)\n",
    "        percentile = (lst[idx] + lst[idx +1 ])/2\n",
    "        \n",
    "    return percentile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now calculate percentiles for any given $p$.\n",
    "\n",
    "For example: let's calculate the $15^{th}$ percentile by using our funtion."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "percentile(sales, 85)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation using `numpy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In practice researchers or data analysts wouldn't implement these \"algorithms\" from scratch. Instead modules that provide such functionalities are used. \n",
    "\n",
    "We can easily calculate percentiles using a new module:\n",
    "\n",
    "- `import numpy`: numpy is one of the most fundamental packages providing functionality for scientific computing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales = load_sales()\n",
    "percentile = np.percentile(sales, q=85)\n",
    "percentile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the result using numpy differs from previous result. This is due to the percentile function applying a different method for interpolation. We can fix this by providing an additional parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.percentile(sales, q=85, interpolation=\"higher\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.percentile? # np.percentile??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Quantiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In statistics it is often helpful to divide data into equally sized groups called **quantiles**.\n",
    "\n",
    "The include quartiles (dividing the observation into 4 groups) and percentiles (100 groups)#:\n",
    "\n",
    "- $Q_1$ = lower quartile, $25^{th}$ percentile\n",
    "- $Q_2$ = median, $50^{th}$ percentile (equal to the median)\n",
    "- $Q_3$ = upper quartile, $75^{th}$ percentile\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference between the upper and lower quartiles is called the **interquartile range** and measures the spread of a distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `numpy` you can easily calculate quartiles as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales = load_sales()\n",
    "quartiles = np.quantile(sales,q=[0.25,0.5,0.75])\n",
    "quartiles"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use other quantiles, e.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "quantiles = np.quantile(sales, q=[0.1,0.5,0.9])\n",
    "quantiles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interquartile_range = quantiles[-1] - quantiles[0]\n",
    "interquartile_range"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Measures of Variability"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Standard deviation & variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **standard deviation** measures the average deviation from the mean and is defined as follows:\n",
    "\n",
    "**Sample** standard deviation:\n",
    "\n",
    "$$s = \\sqrt{\\frac{1}{n-1}\\sum_{i=1}(x_i - \\bar{x})^2}$$\n",
    "\n",
    "**Population** standard deviation:\n",
    "\n",
    "$$\\sigma = \\sqrt{\\frac{1}{n}\\sum_{i=1}(x_i - \\mu)^2}$$\n",
    "\n",
    "where $\\mu$ is the population mean and $\\bar{x}$ is the sample mean.\n",
    "\n",
    "The **sample variance** is the given by $s^2$, the **population variance** is the given by $\\sigma^2$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def std(lst, dof=0):\n",
    "    n = len(lst)\n",
    "    mean = sum(lst)/n\n",
    "    dev = sum([(x - mean)**2 for x in lst])\n",
    "    return (1/(n - dof) * dev)**(1/2) \n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "std_pop = std(sales)\n",
    "std_smpl = std(sales, dof=1)\n",
    "std_pop, std_smpl"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going forward we will use `numpy` for many of our calculations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.std(sales), np.std(sales,ddof=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## $z$-Scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Definition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Represents the number of standard deviation an observation is above or below the mean. \n",
    "\n",
    "The **$z$-score** for the $i_{th}$ observation of (sample) variable $x$ is defined as:\n",
    "\n",
    "$$ \\text{$z$-score of $x_i$}= \\dfrac{x_i - \\bar{x}}{s_x} $$\n",
    "\n",
    "where $\\bar{x}$ and $s_x$ are the mean and the standard devation of x, respectively.\n",
    "\n",
    "The $z$-score standardizes a variable so its unit of measurement no longer matters. To this end, it is not sensitive to how the variable is scaled and/or shifted.\n",
    "\n",
    "**Example**: if the $z$-score of a particular observation equals 2.5, the observation is 2.5 standard deviations above the mean. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales = load_sales()\n",
    "mean = np.mean(sales)\n",
    "std = np.std(sales)\n",
    "z_scores = (sales - mean) / std\n",
    "z_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def zscore(lst):\n",
    "    return (lst - np.mean(lst))/np.std(lst)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zscore(sales)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# New Python concepts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## New functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "list.`sort()`: \n",
    "\n",
    "- sorts a list in ascending order and can be applied to any list object. \n",
    "- is an inplace method meaning the method doesn't return a new list, but sorts the existing list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names = [\"Denise\",\"Beatrix\",\"Adam\", \"Zack\"]\n",
    "names.sort() # or names.sort(reverse=False)\n",
    "names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "names.sort(reverse=True)\n",
    "names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list.sort?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`float.is_integer()`:\n",
    "\n",
    "- checks if a float (a number with digits) is a whole number (e.g. 3.0)\n",
    "- can only be applied to numbers with digits"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number = 3.0\n",
    "number.is_integer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number = 5.9\n",
    "number.is_integer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "number = 3\n",
    "#number.is_integer() # Does not work as an integer object doesn't have a method \"is_integer()\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparison operators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With these operators we can compare two variables. In Python you can use the following comparison operators\n",
    "\n",
    "| Operator | Name                     |\n",
    "|:--------:|--------------------------|\n",
    "|    ==    | Equal                    |\n",
    "|    !=    | Not equal                |\n",
    "|     >    | Greater than             |\n",
    "|     <    | Less than                |\n",
    "|    >=    | Greater than or equal to |\n",
    "|    <=    | Less than or equal to    |\n",
    "\n",
    "Comparison operators return `true` if comparison is true else they return `false`.\n",
    "\n",
    "Let's look at some examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a = 10\n",
    "b = 12\n",
    "a > b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "a != b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "name1 = \"Felix\"\n",
    "name2 = \"Riza\"\n",
    "len(name1) < len(name2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n1 = [1,2,3,4]\n",
    "n2 = [4,3,2,1]\n",
    "sum(n1) == sum(n2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## If condition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If statements are very useful as the run code depending on a test condition. \n",
    "\n",
    "So syntax in Python is as follows:\n",
    "\n",
    "```Python\n",
    "if condition:\n",
    "    do something\n",
    "```\n",
    "\n",
    "We can also write:\n",
    "\n",
    "```Python\n",
    "if condition:\n",
    "    do something\n",
    "else: \n",
    "    do something differently\n",
    "```\n",
    "\n",
    "If we want to test more than one condition we can do this with the `elif` keyword\n",
    "\n",
    "```Python\n",
    "if condition1:\n",
    "    do something\n",
    "elif condition2: \n",
    "    do something differently\n",
    "else:\n",
    "    do something by default\n",
    "```\n",
    "\n",
    "**Example**: let's write a function that tests if a number is big or not. Any number above $100$ is defined as big"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def is_big(number):\n",
    "    if number > 100:\n",
    "        print(number, \"is a big number\")\n",
    "    else: \n",
    "        print(number, \"is a small number\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_big(12)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "is_big(101)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing modules"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the functionality we are using has been programmed and packaged into a specific module. We can access a module with the keyword `import`. After importing a module we can then use it and access the associated objects and methods.\n",
    "\n",
    "```Python\n",
    "import module_name\n",
    "\n",
    "module_name.module_function()\n",
    "\n",
    "```\n",
    "\n",
    "> **Note**: importing a module requires most modules to be downloaded and installed first. This will be explained in more detail in due course\n",
    "\n",
    "**Example**: below we are importing the module`statistics` which provides basic functionality for statistical analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the above cell is executed we can use the module. Here we are using the `mean` function from statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "visitors = [80000, 75000, 78000,80500, 80140]\n",
    "statistics.mean(visitors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**: below we are importing `math` which provides basic math functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "math.pi # calculates pi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "math.ceil(3.4) # rounds up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Example**: `numpy`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's introduce one of the most important modules for data analysis: `numpy`. \n",
    "\n",
    "Numpy [see here](https://numpy.org/) is one of the most fundamental packages of python and provides a vast amount of functionality for scientific computing and is used by researchers throughout the world to facilitate there research (e.g. numpy played an important role in creating the first picture of a black hole; [see here](https://numpy.org/case-studies/blackhole-image/))\n",
    "\n",
    "Many other fundamental modules are built on or rely heavily on numpy (e.g. most of the modules we will use during our course).  \n",
    "\n",
    "For more background please also refer to the recently published paper in Nature ([paper](https://www.nature.com/articles/s41586-020-2649-2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "numbers = [1,2,4,5,6]\n",
    "np.sum(numbers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.percentile(numbers, q=85, interpolation=\"higher\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will gradually introduce some additional modules that are integral for doing data analysis in python. Some of the most important package we will learn about are:\n",
    "\n",
    "- `pandas`: module for tabular data and dataframes\n",
    "- `matplotlib`: module for visualizations\n",
    "- `scipy`: module for scientific analysis\n",
    "- `sklearn`: module for machine learning"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc-autonumbering": true,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}