Week2_3-ForLoops.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>\n",
       "<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>\n",
       "\n",
       "<style>\n",
       "\n",
       "@font-face {\n",
       "    font-family: \"Computer Modern\";\n",
       "    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
       "}\n",
       "\n",
       "#notebook_panel { /* main background */\n",
       "    background: #ddd;\n",
       "    color: #000000;\n",
       "}\n",
       "\n",
       "\n",
       "\n",
       "/* Formatting for header cells */\n",
       ".text_cell_render h1 {\n",
       "    font-family: 'Philosopher', sans-serif;\n",
       "    font-weight: 400;\n",
       "    font-size: 2.2em;\n",
       "    line-height: 100%;\n",
       "    color: rgb(0, 80, 120);\n",
       "    margin-bottom: 0.1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\t\n",
       ".text_cell_render h2 {\n",
       "    font-family: 'Philosopher', serif;\n",
       "    font-weight: 400;\n",
       "    font-size: 1.9em;\n",
       "    line-height: 100%;\n",
       "    color: rgb(200,100,0);\n",
       "    margin-bottom: 0.1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\t\n",
       "\n",
       ".text_cell_render h3 {\n",
       "    font-family: 'Philosopher', serif;\n",
       "    margin-top:12px;\n",
       "    margin-bottom: 3px;\n",
       "    font-style: italic;\n",
       "    color: rgb(94,127,192);\n",
       "}\n",
       "\n",
       ".text_cell_render h4 {\n",
       "    font-family: 'Philosopher', serif;\n",
       "}\n",
       "\n",
       ".text_cell_render h5 {\n",
       "    font-family: 'Alegreya Sans', sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 16pt;\n",
       "    color: grey;\n",
       "    font-style: italic;\n",
       "    margin-bottom: .1em;\n",
       "    margin-top: 0.1em;\n",
       "    display: block;\n",
       "}\n",
       "\n",
       ".text_cell_render h6 {\n",
       "    font-family: 'PT Mono', sans-serif;\n",
       "    font-weight: 300;\n",
       "    font-size: 10pt;\n",
       "    color: grey;\n",
       "    margin-bottom: 1px;\n",
       "    margin-top: 1px;\n",
       "}\n",
       "\n",
       ".CodeMirror{\n",
       "        font-family: \"PT Mono\";\n",
       "        font-size: 100%;\n",
       "}\n",
       "\n",
       "</style>\n",
       "\n"
      ],
      "text/plain": [
       "HTML{ASCIIString}(\"<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>\\n<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>\\n<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>\\n<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>\\n<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>\\n\\n<style>\\n\\n@font-face {\\n    font-family: \\\"Computer Modern\\\";\\n    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\\n}\\n\\n#notebook_panel { /* main background */\\n    background: #ddd;\\n    color: #000000;\\n}\\n\\n\\n\\n/* Formatting for header cells */\\n.text_cell_render h1 {\\n    font-family: 'Philosopher', sans-serif;\\n    font-weight: 400;\\n    font-size: 2.2em;\\n    line-height: 100%;\\n    color: rgb(0, 80, 120);\\n    margin-bottom: 0.1em;\\n    margin-top: 0.1em;\\n    display: block;\\n}\\t\\n.text_cell_render h2 {\\n    font-family: 'Philosopher', serif;\\n    font-weight: 400;\\n    font-size: 1.9em;\\n    line-height: 100%;\\n    color: rgb(200,100,0);\\n    margin-bottom: 0.1em;\\n    margin-top: 0.1em;\\n    display: block;\\n}\\t\\n\\n.text_cell_render h3 {\\n    font-family: 'Philosopher', serif;\\n    margin-top:12px;\\n    margin-bottom: 3px;\\n    font-style: italic;\\n    color: rgb(94,127,192);\\n}\\n\\n.text_cell_render h4 {\\n    font-family: 'Philosopher', serif;\\n}\\n\\n.text_cell_render h5 {\\n    font-family: 'Alegreya Sans', sans-serif;\\n    font-weight: 300;\\n    font-size: 16pt;\\n    color: grey;\\n    font-style: italic;\\n    margin-bottom: .1em;\\n    margin-top: 0.1em;\\n    display: block;\\n}\\n\\n.text_cell_render h6 {\\n    font-family: 'PT Mono', sans-serif;\\n    font-weight: 300;\\n    font-size: 10pt;\\n    color: grey;\\n    margin-bottom: 1px;\\n    margin-top: 1px;\\n}\\n\\n.CodeMirror{\\n        font-family: \\\"PT Mono\\\";\\n        font-size: 100%;\\n}\\n\\n</style>\\n\\n\")"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Setting up a custom stylesheet in IJulia\n",
    "file = open(\"style.css\") # A .css file in the same folder as this notebook file\n",
    "styl = readall(file) # Read the file\n",
    "HTML(\"$styl\") # Output as HTML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1> \"for\" loops and date-time formats: converting string values into usable time data </h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>In this lecture</h2>\n",
    "\n",
    "- [Outcome](#Outcome)\n",
    "- [Converting a date string to DateTime format](#Converting-a-date-string-to-DateTime-format)\n",
    "- [\"for\" loops](#\"for\"-loops)\n",
    "- [Converting column 1 DateTime type](#Converting-column-1-DateTime-type)\n",
    "- [Creating data giving time in days since 22 March 2014](#Creating-data-giving-time-in-days-since-22-March-2014)\n",
    "- [Exporting the converted data](#Exporting-the-converted-data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to the top](#In-this-lecture)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Outcome</h2>\n",
    "\n",
    "After this lecture, you will be able to\n",
    "- Convert a string with date data to Julia's DateTime format\n",
    "- Use Dates.datetime2rata() to calculate the number of days between two DateTime dates\n",
    "- Use a for loop to convert the dates of the epidemic data to days since 22 March 2014\n",
    "- Use writedlm to save the converted data as .csv file\n",
    "\n",
    "[Back to the top](#In-this-lecture)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by reading in the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "54x9 Array{Any,2}:\n",
       " \"25 Nov 2015\"  28637  11314  3804  2536  …  4808     14122     3955   \n",
       " \"18 Nov 2015\"  28634  11314  3804  2536     4808     14122     3955   \n",
       " \"11 Nov 2015\"  28635  11314  3805  2536     4808     14122     3955   \n",
       " \"4 Nov 2015\"   28607  11314  3810  2536     4808     14089     3955   \n",
       " \"25 Oct 2015\"  28539  11298  3806  2535     4808     14061     3955   \n",
       " \"18 Oct 2015\"  28476  11298  3803  2535  …  4808     14001     3955   \n",
       " \"11 Oct 2015\"  28454  11297  3800  2534     4808     13982     3955   \n",
       " \"27 Sep 2015\"  28388  11296  3805  2533     4808     13911     3955   \n",
       " \"20 Sep 2015\"  28295  11295  3800  2532     4808     13823     3955   \n",
       " \"13 Sep 2015\"  28220  11291  3792  2530     4808     13756     3953   \n",
       " \"6 Sep 2015\"   28147  11291  3792  2530  …  4808     13683     3953   \n",
       " \"30 Aug 2015\"  28073  11290  3792  2529     4808     13609     3953   \n",
       " \"16 Aug 2015\"  27952  11284  3786  2524     4808     13494     3952   \n",
       " ⋮                                        ⋱                            \n",
       " \"9 Aug 2014\"    1835   1011   506   373      323       730      315   \n",
       " \"30 Jul 2014\"   1437    825   472   346      227       574      252   \n",
       " \"23 Jul 2014\"   1201    672   427   319      129       525      224   \n",
       " \"14 Jul 2014\"    982    613   411   310  …   106       397      197   \n",
       " \"2 Jul 2014\"     779    481   412   305       75       252      101   \n",
       " \"17 Jun 2014\"    528    337   398   264       24        97       49   \n",
       " \"27 May 2014\"    309    202   281   186       11        16        5   \n",
       " \"12 May 2014\"    260    182   248   171       11          \"–\"      \"–\"\n",
       " \"1 May 2014\"     239    160   226   149  …    11          \"-\"      \"-\"\n",
       " \"14 Apr 2014\"    176    110   168   108        2          \"–\"      \"–\"\n",
       " \"31 Mar 2014\"    130     82   122    80        2          \"–\"      \"–\"\n",
       " \"22 Mar 2014\"     49     29    49    29         \"–\"       \"–\"      \"–\""
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wikiEVDraw = readdlm(\"wikipediaEVDraw.csv\", ',')  # "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to the top](#In-this-lecture)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Converting a date string to DateTime format</h2>\n",
    "\n",
    "Remarkably enough, data on dates and times are among the fiddliest things a data scientist has to deal with. There is a huge number of different ways in which such data are reported, and moreover there are conflicting standards of how to deal irregularities (month lengths aren't all the same, some years are leap years, time zones shift ...).\n",
    "\n",
    "In consequence, every computer language that deals with date-time data has a rich array of  functions to deal with it. In Julia, they are in a package called Dates. Of this package, we will use the functions DateTime() and Dates.datetime2rata().\n",
    "\n",
    "Why does one of them use a full stop and the other not? The answer is that when you start up Julia, only a few of the functions in the package Dates are visible. These functions include Date() but not datetime2rata(). However, we are able to access the other functions by use of the colon. We will talk more about packages in the next lecture.\n",
    "\n",
    "The DateTime() function uses a format string convert string data such as we see in column one into Julia DateTime data.\n",
    "\n",
    "A format string is something one sees in many computation contexts. Here, it tells Julia in what form to expect the data. Looking at the string, it is a number for the day, then space, then an abbreviation for the month, then a space, then a number for the year. The appropriate format string is therefore \"d u y\". These formats have limitations: \"d\" accepts one- and two-digit days (which should always work) and \"y\" accepts two- and four-digit years (which should mostly work), but \"u\" accepts only three-letter abbreviations. Unfortunately, data where the month names otherwise abbreviated are fairly common.\n",
    "\n",
    "Here is an example of how the conversion works\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2015-11-25T00:00:00"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DateTime(wikiEVDraw[1,1], \"d u y\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to the top](#In-this-lecture)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>\"for\" loops</h2>\n",
    "\n",
    "Now we need to do this conversion for every element in column 1 of the matrix. The way to do this is with a \"for\" loop.\n",
    "\n",
    "\"for\" loops are extremely important in computing, and in Julia even more so. This is because many items that are vectorised in languages like Matlab and Python are explicitly computed in \"for\" loops in Julia. It may surprise many of you who know about speeding up computations using vectorisation, but it is frequently the case that a loop in Julia runs *faster* than the equivalent vectorised code.\n",
    "\n",
    "\"for\" loops have a simple structure: the outside is the \"for ... end\" part and the inside is a code block executed repeatedly. Exactly how many times is determined by the the \"for ... end\" part.\n",
    "\n",
    "In the two examples below, we use println() to show the value of the variable over which the \"for\" loop runs. Notice that these values do not have to be a sequence of integers.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "num is now 3\n",
      "num is now 4\n",
      "num is now 5\n",
      "num is now 6\n",
      "num is now 7\n",
      "The value of x is now 23\n",
      "The value of x is now my name is not a name\n",
      "The value of x is now ℵ\n"
     ]
    }
   ],
   "source": [
    "for num = 3:7    # here, the colon is used to specify a range; we will see this again\n",
    "    println(\"num is now $num\")\n",
    "end\n",
    "\n",
    "values = [23, \"my name is not a name\", 'ℵ']      # an array with some rather odd elements\n",
    "for x in values    # a for loop can iterate over an array\n",
    "        println(\"The value of x is now $x\")\n",
    "    end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is important to get the first line of a \"for\" loop exactly right. It has the structure \n",
    "\n",
    "\"variable = iterable\"\n",
    "\n",
    "Here, \"iterable\" is anything that is arranged in a sequence in memory. Not all types are, but they certainly include ranges (created with the colon operator \":\") and any single dimension of an array. The \"=\" is an assignment operator, and it assigns to \"variable\" the values in \"iterable\", one after the other. That is, during each pass through the loop, \"variable\" has the value of exactly one of the items in \"iterable\". "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to the top](#In-this-lecture)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "<h2>Converting column 1 DateTime type</h2>\n",
    "\n",
    "Now we use a \"for\" loop twice. Firstly we create a one-dimensional array containing just column one---it uses array slicing, for conversion to values with DateTime type.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "54-element Array{Any,1}:\n",
       " \"25 Nov 2015\"\n",
       " \"18 Nov 2015\"\n",
       " \"11 Nov 2015\"\n",
       " \"4 Nov 2015\" \n",
       " \"25 Oct 2015\"\n",
       " \"18 Oct 2015\"\n",
       " \"11 Oct 2015\"\n",
       " \"27 Sep 2015\"\n",
       " \"20 Sep 2015\"\n",
       " \"13 Sep 2015\"\n",
       " \"6 Sep 2015\" \n",
       " \"30 Aug 2015\"\n",
       " \"16 Aug 2015\"\n",
       " ⋮            \n",
       " \"9 Aug 2014\" \n",
       " \"30 Jul 2014\"\n",
       " \"23 Jul 2014\"\n",
       " \"14 Jul 2014\"\n",
       " \"2 Jul 2014\" \n",
       " \"17 Jun 2014\"\n",
       " \"27 May 2014\"\n",
       " \"12 May 2014\"\n",
       " \"1 May 2014\" \n",
       " \"14 Apr 2014\"\n",
       " \"31 Mar 2014\"\n",
       " \"22 Mar 2014\""
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col1 = wikiEVDraw[:, 1]  # the colon means all the data in the column, the 1 means the first column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use a \"for\" loop to overwrite the data in the variable col1 with the data in converted using DateTime():"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i = 1:54\n",
    "    col1[i] = DateTime(col1[i], \"d u y\")  # note that this replaces the previous value in col1[i]\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "54-element Array{Any,1}:\n",
       " 2015-11-25T00:00:00\n",
       " 2015-11-18T00:00:00\n",
       " 2015-11-11T00:00:00\n",
       " 2015-11-04T00:00:00\n",
       " 2015-10-25T00:00:00\n",
       " 2015-10-18T00:00:00\n",
       " 2015-10-11T00:00:00\n",
       " 2015-09-27T00:00:00\n",
       " 2015-09-20T00:00:00\n",
       " 2015-09-13T00:00:00\n",
       " 2015-09-06T00:00:00\n",
       " 2015-08-30T00:00:00\n",
       " 2015-08-16T00:00:00\n",
       " ⋮                  \n",
       " 2014-08-09T00:00:00\n",
       " 2014-07-30T00:00:00\n",
       " 2014-07-23T00:00:00\n",
       " 2014-07-14T00:00:00\n",
       " 2014-07-02T00:00:00\n",
       " 2014-06-17T00:00:00\n",
       " 2014-05-27T00:00:00\n",
       " 2014-05-12T00:00:00\n",
       " 2014-05-01T00:00:00\n",
       " 2014-04-14T00:00:00\n",
       " 2014-03-31T00:00:00\n",
       " 2014-03-22T00:00:00"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "col1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Creating data giving time in days since 22 March 2014</h2>\n",
    "\n",
    "\n",
    "Finally, we create the variable \"epidays\". This calls to mind the concept of *epidemic day*, which is simply a way to indicate how long an epidemic has been running. We will assume that the epidemic started on 22 March 2014, with a total of 49 cases, because that is the  first date for which we have data.\n",
    "\n",
    "\n",
    "Note that this is in keeping with the spirit of modelling: we are trying to do the best we can with the data we have. Even when we know that the epidemic has been traced back to a single case in early December 2013, that information is not in the table of data before us. We should not forget about it, but neither should we attempt to include it in the data.\n",
    "\n",
    "The function we use is Dates.datetime2rata(). The \"Rata Die days\" format is a specialised date format we will not discuss here (see https://en.wikipedia.org/wiki/Rata_Die for information). The important thing is that this function, applied to a given date, gives the number of days since 1 January of the year 0001. As follows:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "735927"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Dates.datetime2rata(col1[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Exporting the converted data</h2>\n",
    "\n",
    "\n",
    "We create a function to express the number of days since epidemic day zero, which is the value of col[54] which is of course 22 March 2014.\n",
    "\n",
    "Then we iterate that function with a for loop over all the elements in col1 to create epidays. Note that the variable epidays is created before the start of the loop. This is, in general, good practice: if you know what array you want to fill, then initialise that array before you start filling it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "dayssincemar22(x) = Dates.datetime2rata(x) - Dates.datetime2rata(col1[54])\n",
    "epidays = Array(Int64, 54)\n",
    "for i = 1:54\n",
    "    epidays[i] = dayssincemar22(col1[i])\n",
    "end\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we overwite column 1 of our data array with epidays, and save it using writedlm(). It is a good idea to use a new filename, so that all the work that went into extracting the data from wikipedia is not lost. You never know when you might want the original dates again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "wikiEVDraw[:, 1] = epidays\n",
    "writedlm(\"wikipediaEVDdatesconverted.csv\", wikiEVDraw, ',')  \n",
    "#         note the delimiter ... the Julia default is a tab; to get .csv, we must specify the comma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Back to the top](#In-this-lecture)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.4.6",
   "language": "julia",
   "name": "julia-0.4"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.4.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}