02__Data_Visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><img src='img/ms_logo.jpeg' height=40% width=40%></center>\n",
    "\n",
    "<center><h1>Data Visualization and Plotting</h1></center>\n",
    "\n",
    "Data by itself is useless; without any context, we have no way to understand it.  With different data science techniques, we can extract information from the data, and use this to drive decision making.\n",
    "\n",
    "The main way we've investigated data so far is with descriptive statistics.  Today, we're going to add data visualization to our toolbox, and see a real example of where descriptive statistics without accompanying visualization can be misleading.  \n",
    "\n",
    "Let's start by importing the first data set we'll be working with, and investigating it with some descriptive statistics.  \n",
    "\n",
    "The data set is a CSV, and can be found in the datasets subdirectory.  \n",
    "\n",
    "**Use pandas to read this data set into a dataframe below.  Call .describe() on the dataset to print the summary statistics.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "np.random.seed(1547)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Read in AQ_Data.csv as a dataframe.\n",
    "df = None\n",
    "\n",
    "# Call .describe() on the dataframe. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 4 different data subsets of data contained within this csv file.  The x and y cooridates for each point is denoted by the number following it--the points for the first data set are (x1, y1), (x2, y2) for the second data set, etc. \n",
    "\n",
    "The summary statistics suggest that these datasets are all almost exactly the same!  The mean and standard deviation of the points in each data set are essentially identical.  There are several other identical statistics:\n",
    "\n",
    "1.  The line of best fit for each data set is y = 3.0 + 0.5x\n",
    "1.  For each data set, the correlation between x and y values are 0.816\n",
    "1.  For each data set, the coefficient of determination is 0.67\n",
    "\n",
    "Looking purely at the summary statistics, the evidence is pretty clear.  These data sets are basically identical.  Let's graph them and confirm this suspicion. Run the cell below to plot each data set side-by-side.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "TypeError",
     "evalue": "'NoneType' object is not subscriptable",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTypeError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-69b0eb284e28>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# Run this cell--you do not need to write any code!\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mx\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'x1'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0my1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'y1'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0my2\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'y2'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mTypeError\u001b[0m: 'NoneType' object is not subscriptable"
     ]
    }
   ],
   "source": [
    "# Run this cell--you do not need to write any code!\n",
    "\n",
    "x = df['x1']\n",
    "y1 = df['y1']\n",
    "y2 = df['y2']\n",
    "y3 = df['y3']\n",
    "x4 = df['x4']\n",
    "y4 = df['y4']\n",
    "\n",
    "def fit(x):\n",
    "    return 3 + 0.5 * x\n",
    "\n",
    "xfit = np.array([np.min(x), np.max(x)])\n",
    "\n",
    "plt.subplot(221)\n",
    "plt.plot(x, y1, 'bo', xfit, fit(xfit), 'r-', lw=2)\n",
    "plt.axis([2, 20, 2, 14])\n",
    "plt.setp(plt.gca(), xticklabels=[], yticks=(4, 8, 12), xticks=(0, 10, 20))\n",
    "plt.text(3, 12, 'I', fontsize=20)\n",
    "\n",
    "plt.subplot(222)\n",
    "plt.plot(x, y2, 'bo', xfit, fit(xfit), 'r-', lw=2)\n",
    "plt.axis([2, 20, 2, 14])\n",
    "plt.setp(plt.gca(), xticks=(0, 10, 20), xticklabels=[],\n",
    "         yticks=(4, 8, 12), yticklabels=[], )\n",
    "plt.text(3, 12, 'II', fontsize=20)\n",
    "\n",
    "plt.subplot(223)\n",
    "plt.plot(x, y3, 'bo', xfit, fit(xfit), 'r-', lw=2)\n",
    "plt.axis([2, 20, 2, 14])\n",
    "plt.text(3, 12, 'III', fontsize=20)\n",
    "plt.setp(plt.gca(), yticks=(4, 8, 12), xticks=(0, 10, 20))\n",
    "\n",
    "plt.subplot(224)\n",
    "xfit = np.array([np.min(x4), np.max(x4)])\n",
    "plt.plot(x4, y4, 'bo', xfit, fit(xfit), 'r-', lw=2)\n",
    "plt.axis([2, 20, 2, 14])\n",
    "plt.setp(plt.gca(), yticklabels=[], yticks=(4, 8, 12), xticks=(0, 10, 20))\n",
    "plt.text(3, 12, 'IV', fontsize=20)\n",
    "\n",
    "# verify the stats\n",
    "pairs = (x, y1), (x, y2), (x, y3), (x4, y4)\n",
    "for x, y in pairs:\n",
    "    print('mean=%1.2f, std=%1.2f, r=%1.2f' % (np.mean(y), np.std(y),\n",
    "          np.corrcoef(x, y)[0][1]))\n",
    "\n",
    "plt.show()\n",
    "\n",
    "# Code pulled from Matplotlib's tutorial on Anscombe's Quartet:\n",
    "# https://matplotlib.org/gallery/specialty_plots/anscombe.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><h2>WAIT, WHAT?</h2></center>\n",
    "<br>\n",
    "<br>\n",
    "<center><img src='http://www.reactiongifs.com/r/2013/11/betty-faint.gif'></center>\n",
    "\n",
    "Our summary statistics could not have been more wrong! When we graph these data sets, can see that they are qualititatively very, very different:\n",
    "\n",
    "1. Plot 1 is linear\n",
    "1. Plot 2 is non-linear.\n",
    "1. Plot 3 is mostly linear, with an outlier skewing the line of best fit.  \n",
    "1. Plot 4 has little relationship at all--all coordinates but 1 have the exact same X value, regardless of y value!\n",
    "\n",
    "## Anscombe's Quartet\n",
    "\n",
    "The misleading similarities of these data sets are no accident. These data sets are the famous [Anscombe's Quartet](https://en.wikipedia.org/wiki/Anscombe%27s_quartet) (well, famous to statisticians and data scientists, anyway).  This data set was designed by the statistician Francis Anscombe in 1973 to illustrate how misleading summary statistics can be without an accompanying visualization.  Although the numbers tell us the data sets are the same, our eyes can easily see this is not the case.  \n",
    "\n",
    "This brings us to the main lesson Anscombe has so elegantly taught us: **use visualizations whenever possible!**.  \n",
    "<br>\n",
    "<br>\n",
    "\n",
    "<center><h1>Visualizing Data with Matplotlib</h1></center>\n",
    "\n",
    "For the second part of this lesson, we'll explore the different visualizations available in python's most popular data visualization library, [Matplotlib](https://matplotlib.org/index.html).  \n",
    "\n",
    "Matplotlib is a tried-and-true python framework that is one of the first tools data scientists reach for when working with a data set.  Data visualization is a crucial part of the data science process, whether we're doing our initial exploration to get a feel for our data in the beginning of a project, or communicating the results of our experiment to stakeholders in a clear, intuitive way.  \n",
    "\n",
    "### Which graph should I Use?\n",
    "\n",
    "Before we can visualize our data, we'll need to select the type of graph we'll want to use.  There are three main types of graphs we can use:\n",
    "\n",
    "1.  Histograms/bar charts\n",
    "1.  Scatterplots\n",
    "1.  Time Series\n",
    "\n",
    "These types of graphs are the most common (and most useful!) ways to visualize data. Luckily, pandas and matplotlib are designed to fit together effortlessly.  Let's code up some examples of each chart!\n",
    "\n",
    "\n",
    "# Histograms/Bar Charts\n",
    "\n",
    "Histograms are used to represent counts.  If the data you're trying to represent is categorical, or have repeated values across a dimension of your data set, then a histogram is the way to go.  \n",
    "\n",
    "Visualizing histograms is such a common task in pandas that they've actually built it right into the pandas library, eliminating extra code that would normally be needed to use matplotlib.  Every dataframe in pandas has access to the `.hist()` method, which will visualize the data in a histogram.  \n",
    "\n",
    "It's common to mix up bar charts and histograms.  We're going to build one of each to help exemplify the differences, and examine the best use case for each.  \n",
    "\n",
    "**TASK: Use the df.hist( ) method to create a histogram for the sample data (provided below)**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sample_data = {\n",
    "    'one': [1],\n",
    "    'two': [1],\n",
    "    'fish': [4],\n",
    "    'red': [1],\n",
    "    'blue': [1]\n",
    "}\n",
    "\n",
    "# Create a dataframe by calling pd.Dataframe() and passing the sample_data dictionary as the first argument.\n",
    "hist_df = None\n",
    "\n",
    "# To create the histogram, we'll have to specify that we're calling it on the first (and only) row of data in \n",
    "# the dataframe.  To do this, we use .iloc[0].  We can chain the hist() method onto the iloc[0] method.\n",
    "#  \n",
    "# Call .iloc[0].hist() on the hist_df object.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If this looks different than you were expecting, it's because you were thinking of a bar chart instead.  Histograms show us the number of times a value shows up.  In this case, 'one', 'two', 'red', and 'blue' all have values of 1, while only 'fish' has a value of 4.  The histogram has binned these examples to show us the total number of times each number shows up in the data set.  In this context, the graph above makes sense--the '1' column is at 4, and the '4' column is at 1.  \n",
    "\n",
    "A **Bar Chart** shows us separate values for different categories, side-by-side.  Let's create a bar chart of the same data and see how it looks.  \n",
    "\n",
    "**TASK: Create a bar chart using the .plot.bar() method on a new dataframe.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create another dataframe called bar_df using pd.Dataframe. Pass in sample_data as the argument (this is already \n",
    "# in memory from the last cell ran, so you do NOT need to copy and paste sample data into this cell.)\n",
    "\n",
    "bar_df = None\n",
    "\n",
    "# To create the chart, access the plot object stored in the dataframe's .plot attribute, and call this object's .bar() \n",
    "# method.  This can be chained together as .plot.bar()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This looks more like what we expected! The chart color-codes each word differently for us, and even provides a nice key in the top-right corner for us. \n",
    "\n",
    "**TASK:  In your own words, how would you explain when you should use a histogram, versus when you should use a bar chart?**\n",
    "\n",
    "ANSWER:\n",
    "\n",
    "{WRITE ANSWER HERE!}\n",
    "\n",
    "\n",
    "<center><h2>Scatterplots</h2></center>\n",
    "\n",
    "Scatterplots are one of the simplest, but most effective visualizations out there! The idea behind a scatterplot is intuitive--make one column of the dataframe your x coordinate, make another one your y coordinate, and then plot everything to see how it looks.  \n",
    "\n",
    "Scatterplots are also easy to build use pandas/matplotlib.  \n",
    "\n",
    "**TASK: Create a scatterplot using the sample data provided below.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# We'll use the make_blobs function from Scikit-Learn to generate dummy data with a few clusters in it.  \n",
    "from sklearn.datasets import make_blobs\n",
    "X, _ = make_blobs()\n",
    "scatter_df = pd.DataFrame(X, columns=['x', 'y'])\n",
    "\n",
    "\n",
    "# To create the scatter plot, access the object stored in .plot, and then call the .scatter() method. You can\n",
    "# chain these together, as you did with the bar chart example.  In the .scatter() method, you'll need to pass in 'x'\n",
    "# and 'y' as positional arguments to tell the scatter plot which points belong on each axis.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was easy!  The scatterplot is a great way to get a quick, intuitive feel for our data set.  In this case, we can see that there are three clusters in our data.  In a real data science project, this is the type of thing that would likely warrant more investigation.  If we didn't visualize our data as a scatterplot, we might never have noticed this!\n",
    "\n",
    "\n",
    "<center><h2>Line Graphs/Time Series</h2></center>\n",
    "\n",
    "Line Graphs are most commonly used for Time Series analysis--that is, tracking how data changes over a time interval. A Time Series is a just a line graph where the x-axis is a sequence of datetimes, starting at the left and progressing as you move to the right.  Line graphs are just like a scatterplot, but with a continuous line drawn between the points, starting at the leftmost x value and connecting to each sequential x value.  \n",
    "\n",
    "Line graphs and Time Series plots are one the most important tools used in the financial world.  For our example chart, we're going to graph the price of the Ethereum cryptocurrency.  (If you're not familiar with ethereum, its a very popular cryptocurrency similar to bitcoin.  It's primarily used for HODLing, until it can traded for a lamborghinis on the moon).  \n",
    "\n",
    "<center><img src='https://i.redd.it/3zrf0i340iez.gif' height=25% width=25%></center>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pd' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-78aa7d5dd5e5>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m# This\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0meth_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'datasets/ethereum_price_data.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0meth_df\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0meth_df\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;31m# call the .plot method on eth_df.  Be sure to pass in the \"Date\" column name (as a string) for the x keyword,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'pd' is not defined"
     ]
    }
   ],
   "source": [
    "# This \n",
    "eth_df = pd.read_csv('datasets/ethereum_price_data.csv')\n",
    "eth_df = eth_df.iloc[::-1]\n",
    "\n",
    "# call the .plot method on eth_df.  Be sure to pass in the \"Date\" column name (as a string) for the x keyword, \n",
    "# and the \"Close\" column name (also a string) for the y keyword.  Give the graph a label with the title with the \n",
    "# title keyword, and pass in a in the tuple (12,5) for the figsize keyword (this will set the size of the graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That was pretty easy to do, and the graph is very easy to understand.  Note that we had to add a few keyword arguments to make the graph large enough to read without the X-axis items overlapping.  It's easy to adjust the looks of any plot with keyword arguments--after all, `Dataframe.plot` is just a wrapper around `matplotlib.pyplot`.  For a full list of keywords you can use to modify your plot, check out the [documentation!](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.plot.html#pandas.DataFrame.plot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><h1>Challenge Section</h1></center>\n",
    "\n",
    "For the remainder of this lesson, you'll be using the Titanic data set to create several visualizations that communicate something interesting.  For instance, can you visualize survival rates of men vs. women?  How about by age? Fare price? \n",
    "\n",
    "To complete this section, you'll need to:\n",
    "\n",
    "1.  Think of at least 3 interesting questions to investigate in the data set.  \n",
    "1.  Select the most appropriate graph type to represent the data relevant to each question.  \n",
    "1.  Create a visualization that attempts to answer the questions you have posed!\n",
    "\n",
    "**TASK: Create at least 3 visualizations related to the Titanic data set.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Read in titanic.csv, which is contained within the datasets folder.\n",
    "\n",
    "titanic_df = None"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}