3_Data_Collection.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:100923ce92e594169a2f1f4c3bae3ff28540d70db17c2a1bb95481c3f4dc149b"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Data Collection\n",
      "===\n",
      "Up until now, the data that you've been working with has been given to in the form of a text file, a database, or in a python pickle. In this module, we'll learn a few techniques for writing our own python scripts for collecting data from the internet. \n",
      "\n",
      "Collecting data from the internet can be done in one of two ways:\n",
      "1. through an API (Application Program Interface)\n",
      "2. by parsing html\n",
      "\n",
      "The API route is always preferable as it is (usually) sactioned by whomever is hosting the data, which allows them to dictate terms of use and monitor and restrict how their service is used. An access token (or \"key\") is typically required to access data from an official API.\n",
      "\n",
      "Not all websites provide a public API for accessing data. In those cases, you may have to write a custom HTML parser. HTML is the language that the static components of webpages are written in. This code is usually well structured and it is often times not difficult to write a short script to find the elements of a interest in the HTML.\n",
      "\n",
      "Getting Data via an API\n",
      "---\n",
      "Each API is different as is the process getting an API key. Usually getting an API key is as simple as creating an account and registering for a key. Almost all companies that offer a public API, also offer documentation on how to use their API and how to get a key. Here's Instagram's Developer's page: https://instagram.com/developer/\n",
      "\n",
      "Go ahead and go there and register a new client.  \n",
      "\n",
      ">**Note:** Most API users have their own web application and are using the Instagram API to allow the users of their web app to integrate their instagram data with their app. The client ID and client secret are specific to the web application and the access token is specific to the user of the web application. You can think of the access token as a custom username and password that the client can use on behalf of its users to get access to certain data that is only available to a user logged into Instagram. Something similar happens everytime you use facebook or google to \"login\" to a third party site. The third party side gets an access token from facebook or google and can use that to get your name, birthdate, gender or any other information that you've agreed to give it access to.\n",
      "Although we won't be making any requests on anybody elses behalf, we'll still need an access token to access Instagram data as though we were logged in.\n",
      "\n",
      "When filling out the form to register your client. You can use a personal webpage or any publicly available webpage for the \"Website URL\" and \"Redirect URI\". Before clicking the \"Register\" button, click the \"Security\" tab and uncheck the box that says \"Disable implicit OAuth\".  After you've registered write down your Client Id. Now click on the \"Autentecation\" tab on the left and follow the \"Client-Side (Implicit) Authentication\" directions to get an access token.\n",
      "\n",
      ">**Note:** Where the directions say \"Direct your user to our authorization URL\" you can copy and paste that url in your browser. Make sure you replace \"CLIENT-ID\" with your client id and \"REDIRECT-URI\" with whatever redirect url you chose when registering your client. When you click the \"authorize\" button, you'll get redirected to your \"Redirect URI\", but the url will include something like \"#access_token=703990239.f1d54a3.8cf94c42373e43a2951bce01d7ac0bf1\". Everything after the \"=\" is your access_token.\n",
      "\n",
      "Once you finish you should receive a client id. Replace the strings below with your client id and access token before continuing with the rest of this module."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "client_id = \"replace this string with your client id\"\n",
      "access_token = \"replace this string with your access token\""
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 3
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In order to get data from Instagram, we'll be making \"requests\" to their API. We'll be making our requests through python, most you could also make a request directly from your browser. Your browser is in fact making dozens or even hundreds of requests every time it loads a web page. In the cases where your browser loads a web page, it's converting HTML into well formatted human-readable content. When we make requests to an API we'll usually get back content that was designed for a computer to read. Paste the following url (replacing YOUR-ACCESS-TOKEN with your access token) into your browser.\n",
      "\n",
      "````\n",
      "https://api.instagram.com/v1/users/search?q=adamhajari&access_token=YOUR-ACCESS-TOKEN\n",
      "````\n",
      "\n",
      "What you get back is a JSON object that contains metadata for the Instagram user `adamhajari`. We breifly saw JSON objects in the previous module. They're identical in structure to python dictionaries, and python comes with a library for converting JSON strings into a dictionary.\n",
      "\n",
      "Let's make that same API request, now using python's `requests` library:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import requests\n",
      "url = \"https://api.instagram.com/v1/users/search?q=adamhajari&access_token=%s\"%access_token\n",
      "response = requests.get(url)\n",
      "response.text"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 4,
       "text": [
        "u'{\"meta\":{\"code\":200},\"data\":[{\"username\":\"adamhajari\",\"profile_picture\":\"https:\\\\/\\\\/instagramimages-a.akamaihd.net\\\\/profiles\\\\/profile_703880175_75sq_1385171062.jpg\",\"id\":\"703880175\",\"full_name\":\"Adam Hajari\"}]}'"
       ]
      }
     ],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The response object has other metadata about the response (success/error codes, headers, etc.), but we're only interested in the content (or \"text\") of the response. As we saw before, the response text is a JSON string. We'll use python's `json` library to turn that string into a dictionary"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import json\n",
      "response_data = json.loads(response.text)\n",
      "response_data"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 5,
       "text": [
        "{u'data': [{u'full_name': u'Adam Hajari',\n",
        "   u'id': u'703880175',\n",
        "   u'profile_picture': u'https://instagramimages-a.akamaihd.net/profiles/profile_703880175_75sq_1385171062.jpg',\n",
        "   u'username': u'adamhajari'}],\n",
        " u'meta': {u'code': 200}}"
       ]
      }
     ],
     "prompt_number": 5
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "response.headers"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "{'x-ratelimit-remaining': '4998', 'content-language': 'en', 'content-encoding': 'gzip', 'set-cookie': 'csrftoken=ed8a9d4efc5dbf4c387442999ef0d7ea; expires=Mon, 20-Jun-2016 21:01:39 GMT; Max-Age=31449600; Path=/', 'expires': 'Sat, 01 Jan 2000 00:00:00 GMT', 'vary': 'Cookie, Accept-Language, Accept-Encoding', 'content-length': '175', 'connection': 'keep-alive', 'x-ratelimit-limit': '5000', 'pragma': 'no-cache', 'cache-control': 'private, no-cache, no-store, must-revalidate', 'date': 'Mon, 22 Jun 2015 21:01:39 GMT', 'content-type': 'application/json; charset=utf-8'}"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "which we saw how to work with in the previous module:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "user_id = response_data['data'][0]['id']\n",
      "user_id"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "u'703880175'"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      ">**Note:** The \"data\" is actually a list of dictionaries (with only one element), we need to specify that we want the first element in that list\n",
      "\n",
      "This is the Basic workflow for using python to get data from the internet:\n",
      "1. Make a request and get the content\n",
      "2. convert the content into a parsable python object (like a dictionary)\n",
      "3. get the data of interest and do something with it\n",
      "\n",
      "**Exercise 3.1:**\n",
      "Write a function that takes an Instagram username and returns an instagram user id.\n",
      "\n",
      "\n",
      "###API Parameters\n",
      "Instagram's [recent media user endpoint](https://instagram.com/developer/endpoints/users/#get_users_media_recent) returns recent posts for a given user id. By default, it will return 20 of the most recent posts, but you can supply parameters to override some of the defaults. For instance, if you wanted the last 20 posts from 2014, you include the `max_timestamp` parameter in your request with value 1420070400.\n",
      "\n",
      ">**side note: what is a unix timestamp?**  A unix timestamp is the number of seconds since January 1, 1970. While it's a pretty inconvenient way to represent time from the point-of-view of human readability, you'll often see it used in data storage because it's easier to use in calculations, takes up less space, and is faster to lookup than a datetime string. Python's `datetime` library has a few functions for converting to and from unix time. You can also do these conversion for unix timestamps stored in a dataframe with pandas. And, if you have a single unix timestamp that you're going to lookup just once, you can also do so manually [here](http://www.epochconverter.com).\n",
      "\n",
      "The API parameters are passed in as a suffix to the url. The first parameter is preceded with a `?` and all subsequent parameters are separated by an `&`. All api request require at least one parameter (the access token). A request url using the `max_timestamp` parameter would look like this:\n",
      "````\n",
      "https://api.instagram.com/v1/users/260133476/media/recent/?access_token=703990239.f1d54a3.8cf94c42373e43a2951bce01d7ac0bf1&max_timestamp=1420070400\n",
      "````\n",
      "You can paste this url in your browser to see the results (be sure the use your own access_token). Note that despite what is implied by the instagram api documentation, your request params should be all lowercase.\n",
      "\n",
      ">**PRO-TIP:** There are browser extensions that make it much easier to view json in your browser (everything is well-formatted and you can collapse values). For those of you who use the chrome web browser, [here's the json formatting extentsion that I use](https://chrome.google.com/webstore/detail/jsonview/chklaanhfefbnpoihckbnefhakgolnmc?hl=en). You may have to restart your to start the extension.\n",
      "\n",
      "**Exercise 3.2:** Get the most recent 30 instagram posts by The Alabama Shakes (instagram username: alabama_shakes). Record the id, the link, the number of comments, the number of likes, and the filter. Put the results in a dataframe.\n",
      "\n",
      "###Pagination\n",
      "An API request doesn't always return all availalbe data. For instance, by default you only get the 20 most recent posts from Instagram's recent media endpoint, but even using the `count` param you can only get a maximum of 33 posts per request. One solution to get data for all posts is to use the `max_id` param and pass it the lowest id of your current results and continue to do this (using a `while-loop`) until the API returns an empty data set. If you're lucky, the API results will include a paginiation link that you can follow to get the next page for you. Then to get all of the results you'd just continue to follow the pagination link until there were no more pagination links.\n",
      "\n",
      "**Exercise 3.3:** Get *every* instagram post by the Alabama Shakes and put the same values from above in another dataframe. Determine which was the bands most popular post and what their favorite filter is.\n",
      "\n",
      "###Rate Limits\n",
      "One important consideration when building a crawler that will make lots of API requests is that it adhere to the rate limits (i.e. number of requests per hour) [specified in the documentation](https://instagram.com/developer/limits/). You can do this either pro-actively or reactively. The proactive solution is to put a pause between each request. Instagram has a 5000 requests per hour rate limit which works out to 1 request every 0.72 seconds. You can use the `sleep` function from python's `time` library to implement these pauses:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import time\n",
      "response = requests.get(url)  # make a request\n",
      "time.sleep(0.72)  # sleep for 0.72 seconds\n",
      "response = requests.get(url)  # make the next request request"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 8
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Sometimes an API response will give you feedback too let you know that you've hit their rate-limit. In that case you can take a reactive approach to limiting your requests. Instagram provides this feedback in the form of a response code (which is part of python's response object). When all is well, the code is 200, but if you've exceeded your rate limit, the code will change to 429."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "response.status_code  # code should be 200 if all is well and 429 if we've exceeded our rate limit"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "200"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      ">**Note:** exceeding the rate limit isn't the only thing that can go wrong with an API call and there's a [different status code for each issue you might have](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes). For instance, if we make a request to the recent media endpoint using an invalid user id, we get a 400 status code (bad request). If we make a request to a non-exising endpoint, we get a 404 (Not Found) status code."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "url = \"https://api.instagram.com/v1/users/0/media/recent/?access_token=%s\"%access_token  # 0 is not a valid media id\n",
      "response = requests.get(url)\n",
      "response.status_code"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 11,
       "text": [
        "400"
       ]
      }
     ],
     "prompt_number": 11
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "url = \"https://api.instagram.com/v1/users/this_is_not_and_endpoint/?access_token=%s\"%access_token\n",
      "response = requests.get(url)\n",
      "response.status_code"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 74,
       "text": [
        "404"
       ]
      }
     ],
     "prompt_number": 74
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Instagram provides one additional handy way to proactively address your request rate by providing the number of requests remaining to your access token in the response header."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import requests\n",
      "url = \"https://api.instagram.com/v1/users/search?q=adamhajari&access_token=%s\"%access_token\n",
      "response = requests.get(url)\n",
      "response.headers"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 59,
       "text": [
        "{'x-ratelimit-remaining': '4986', 'content-language': 'en', 'content-encoding': 'gzip', 'set-cookie': 'csrftoken=5dd23dc2158e34cb8547be96f1dedf31; expires=Mon, 13-Jun-2016 15:54:13 GMT; Max-Age=31449600; Path=/', 'expires': 'Sat, 01 Jan 2000 00:00:00 GMT', 'vary': 'Cookie, Accept-Language, Accept-Encoding', 'content-length': '175', 'connection': 'keep-alive', 'x-ratelimit-limit': '5000', 'pragma': 'no-cache', 'cache-control': 'private, no-cache, no-store, must-revalidate', 'date': 'Mon, 15 Jun 2015 15:54:13 GMT', 'content-type': 'application/json; charset=utf-8'}"
       ]
      }
     ],
     "prompt_number": 59
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "response.headers is a dictionary, which makes it easy to get what we need."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "response.headers['x-ratelimit-remaining']  # number of calls left in our 1-hour window"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 65,
       "text": [
        "'4986'"
       ]
      }
     ],
     "prompt_number": 65
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Parsing HTML for data\n",
      "---\n",
      "An official API is always the best way to get data from a host, but sometimes a website either doesn't have an API or, if they do have an API, some data of interest may not be available via an API endpoint. In those cases you may be able to get the raw HTML and use an HTML parsing library to extract the data of interest.\n",
      "\n",
      "We can get the raw HTML, the same we we got API data, with the `requests` library. Let's grab the HTML from the wikipedia page on Conan episodes from 2014."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "url = \"https://en.wikipedia.org/wiki/List_of_Conan_episodes_(2014)\"\n",
      "response = requests.get(url)\n",
      "html_raw = response.text"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 12
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The response text is raw html which is difficult to work with, so we'll use the [beautiful soup](http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html) library to help us parse out the data we're interested in."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from bs4 import BeautifulSoup # To get everything\n",
      "html_soup = BeautifulSoup(html_raw)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 13
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "HTML is a nested set of opening and closing tags which define the page layout end properties. Everything is typically wrapped in html open (<code>&lt;html></code>) and close (<code>&lt;/html></code>) tags. Between <code>&lt;html>&lt;/html></code> there's usually a <code>&lt;head>&lt;/head></code> section where metadata and javascript code lives, and a <code>&lt;body>&lt;/body></code> section where the page content lives. Once you've soupified your html, You can access nested elements as though they were attributes of their parent object. For instance, to get the `head` section of the page you would do:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "head = html_soup.html.head"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "and to get the title element within the `head` section:"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "head.title"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 16,
       "text": [
        "<title>List of Conan episodes (2014) - Wikipedia, the free encyclopedia</title>"
       ]
      }
     ],
     "prompt_number": 16
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "You can also perform a search on a beautiful soup object. Let's say you want to get the Table of Contents element from this wikipedia page. You know that the Table of Contents HTML element has an `id` attribute with value \"toc\"."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "table_of_contents = html_soup.find(id='toc')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 17
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Beautiful soup objects have a few useful methods and attributes. The `text` attribute is a string of all text within the element that would be displayed in a browser, and the `attrs` attibute is a dictionary of html attributes that are a part of the tag (like the `id` or `class`)."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "table_of_contents.attrs  # the table of contents element has 2 attributs, a class (equal to ['toc']) and an 'id' (equal to 'toc')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 18,
       "text": [
        "{'class': ['toc'], 'id': 'toc'}"
       ]
      }
     ],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print table_of_contents.text"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "\n",
        "\n",
        "Contents\n",
        "\n",
        "\n",
        "1 2014\n",
        "\n",
        "1.1 January\n",
        "1.2 February\n",
        "1.3 March\n",
        "1.4 April\n",
        "1.5 May\n",
        "1.6 June\n",
        "1.7 July\n",
        "1.8 August\n",
        "1.9 September\n",
        "1.10 October\n",
        "1.11 November\n",
        "1.12 December\n",
        "\n",
        "\n",
        "2 References\n",
        "\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "If we want to find *all* instances of an element meeting our search criteria we can instead use the `findAll` method. Let's get a list of all months included in the Table of Contents. If you print the `table_of_contents` object, you'll see that each month is wrapped in a list tag (<code>&lt;li></code>) with class attribute 'toclevel-2'."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "month_elements = table_of_contents.findAll('li', {'class':'toclevel-2'})\n",
      "len(month_elements)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 24,
       "text": [
        "12"
       ]
      }
     ],
     "prompt_number": 24
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "As expected, there are 12 month_elements. Let's see what the first month_element looks like"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "month_elements[0]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "<li class=\"toclevel-2 tocsection-2\"><a href=\"#January\"><span class=\"tocnumber\">1.1</span> <span class=\"toctext\">January</span></a></li>"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Each month element has a tag (a link to somewhere else in the page in this case) and two spans. The text of the spans are the table of content number and month. Let's create a lists for the href, number, and month and put the results in a dataframe"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "hrefs = []\n",
      "toc_numbers = []\n",
      "months  = []\n",
      "for element in month_elements:\n",
      "    a_tag = element.a  # a tags in html are used for links within a page or too another page\n",
      "    hrefs.append(a_tag.attrs['href'])  # the href attribute determines where the link goes\n",
      "    toc_number = element.find('span',{'class':'tocnumber'})\n",
      "    toc_numbers.append(toc_number.text)\n",
      "    month = element.find('span',{'class':'toctext'})\n",
      "    months.append(month.text)\n",
      "\n",
      "import pandas as pd\n",
      "pd.DataFrame({'month':months, 'href':hrefs}, index=toc_numbers)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<div style=\"max-height:1000px;max-width:1500px;overflow:auto;\">\n",
        "<table border=\"1\" class=\"dataframe\">\n",
        "  <thead>\n",
        "    <tr style=\"text-align: right;\">\n",
        "      <th></th>\n",
        "      <th>href</th>\n",
        "      <th>month</th>\n",
        "    </tr>\n",
        "  </thead>\n",
        "  <tbody>\n",
        "    <tr>\n",
        "      <th>1.1</th>\n",
        "      <td>#January</td>\n",
        "      <td>January</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.2</th>\n",
        "      <td>#February</td>\n",
        "      <td>February</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.3</th>\n",
        "      <td>#March</td>\n",
        "      <td>March</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.4</th>\n",
        "      <td>#April</td>\n",
        "      <td>April</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.5</th>\n",
        "      <td>#May</td>\n",
        "      <td>May</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.6</th>\n",
        "      <td>#June</td>\n",
        "      <td>June</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.7</th>\n",
        "      <td>#July</td>\n",
        "      <td>July</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.8</th>\n",
        "      <td>#August</td>\n",
        "      <td>August</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.9</th>\n",
        "      <td>#September</td>\n",
        "      <td>September</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.10</th>\n",
        "      <td>#October</td>\n",
        "      <td>October</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.11</th>\n",
        "      <td>#November</td>\n",
        "      <td>November</td>\n",
        "    </tr>\n",
        "    <tr>\n",
        "      <th>1.12</th>\n",
        "      <td>#December</td>\n",
        "      <td>December</td>\n",
        "    </tr>\n",
        "  </tbody>\n",
        "</table>\n",
        "</div>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "            href      month\n",
        "1.1     #January    January\n",
        "1.2    #February   February\n",
        "1.3       #March      March\n",
        "1.4       #April      April\n",
        "1.5         #May        May\n",
        "1.6        #June       June\n",
        "1.7        #July       July\n",
        "1.8      #August     August\n",
        "1.9   #September  September\n",
        "1.10    #October    October\n",
        "1.11   #November   November\n",
        "1.12   #December   December"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Notice that with the `find` and `findAll` methods, we can specified both the tag type (a, span, div, etc.) as well as attribute values. It's worth looking over the [beautiful soup Quick Start](http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html#Quick Start) documentation to see all of the available methods. If you're not very familiar with HTML you might also consider looking over [this tutorial](http://www.w3schools.com/html/) helpful.\n",
      "\n",
      ">**Pro-Tip:** Most web browsers have their own built-in developer tools for investegating all the HTML and other elements that go into displaying a web page. In Chrome you can right click on any element on a page and click on \"Inspect Element\" to look at it's HTML. Use the arrows on the left of each element to expand or collapse in order to show/hide all of its children.\n",
      "\n",
      "**Exercise 3.4:** Use beautiful soup to get the show number (\"No.\"), air date, lists of guests, and list of entertainment guests for every Conan show from 2014. Put the results in a dataframe.  **Bonus:** make the date a datetime object."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}