final.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ODI Queensland workshop - Web Scraping \n",
    "\n",
    "## QUT DMRC - 2015\n",
    "\n",
    "###  Support for multiple pages\n",
    "\n",
    "This notebook scrapes http://www.qld.gov.au/transport/safety/signs/ and saves the data in a dataframe.\n",
    "The script iterates through the webpage structure (structured by the types of signs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# initialise plotting in the notebook\n",
    "%pylab inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import bs4\n",
    "import requests\n",
    "import pandas as pd\n",
    "from os.path import isfile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Initialise global variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# this is the base_url\n",
    "base_url = \"http://www.qld.gov.au/transport/safety/signs/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# columns labels\n",
    "colnames = [\"sign_name\", \"sign_type\", \"description\", \"images\"]\n",
    "\n",
    "# not using these for now - but needed to clean up display of images column\n",
    "images_cols = [\"image_name\", \"url\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# The pages are organised by type of sign.\n",
    "sign_types = [\"regulatory\",\n",
    "              \"hazard\",\n",
    "              \"warning\",\n",
    "              \"route\",\n",
    "              \"instruction\",\n",
    "              \"tourist\",\n",
    "              \"service\",\n",
    "              \"roadworks\"]\n",
    "\n",
    "# if you want to limit the number of pages to scrape, you simply shorten this list - e.g.\n",
    "sign_types = [\"regulatory\", \"hazard\",]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of having a static list of ```sign_types``` we could scrape the list of pages from the menu on the side of the page - and if we did we could keep their long names as well as the url.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### Function definitions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```get_itemlist``` function is unchanged from the previous page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# processes a beautiful_soup data structure and returns new signs in a dataframe\n",
    "def get_itemlist(thesoup):\n",
    "    \n",
    "    # find all the tables on the page\n",
    "    tables = thesoup.findAll('table')\n",
    "    thelist = []\n",
    "\n",
    "    for table in tables:\n",
    "        # find all the table rows\n",
    "        lotsofitems = table.findAll('tr')\n",
    "\n",
    "        # check if the first row contains a 'th' elements (table header)\n",
    "        if lotsofitems[0].find('th'): \n",
    "\n",
    "            # get all header elements\n",
    "            temp = lotsofitems[0].findAll('th')\n",
    "\n",
    "            # check that the table header has the text we expect for the signs table\n",
    "            if temp[0].get_text() == 'Sign' and temp[1].get_text() == 'Meaning':\n",
    "\n",
    "                # print('Traffic sign table found')\n",
    "\n",
    "                # process the table of traffic signs **** THIS IS THE UPDATED SECTION *****\n",
    "                for an_item in lotsofitems[1:]: \n",
    "                    theitem = []\n",
    "\n",
    "                    # sign description & title\n",
    "                    sign_text = an_item.findAll(\"p\")\n",
    "                    description = ''\n",
    "                    for para in sign_text:\n",
    "                        if para.find(\"strong\"):\n",
    "                            # extract sign name (this assumes only the sign name is in bold)\n",
    "                            temptemp = para.find(\"strong\").get_text()\n",
    "                            temptemp = temptemp.split()\n",
    "                            sign_name = \" \".join(temptemp)\n",
    "                        else:\n",
    "                            # extract sign description (may be multiple paragraphs)\n",
    "                            temptemp = para.get_text()\n",
    "                            temptemp = temptemp.split()\n",
    "                            description += \" \".join(temptemp) + '\\n'\n",
    "\n",
    "                    theitem += [sign_name]\n",
    "                    theitem += [sign_type]\n",
    "                    theitem += [description]\n",
    "\n",
    "                    # sign images (may be more than one image per sign name) - save image name & image url\n",
    "                    images = []\n",
    "                    for image in  an_item.findAll(\"img\"):\n",
    "                        # get the image name & image url\n",
    "                        images += [[image.attrs['alt'], image.attrs['src']]]\n",
    "                    theitem += [images]\n",
    "\n",
    "                    thelist += [theitem]\n",
    "\n",
    "#            else:\n",
    "#                print('Different table - with header row:', temp)\n",
    "#        else:\n",
    "#            print('Different table - no header row:', lotsofitems[0])\n",
    "\n",
    "    return pd.DataFrame(thelist,columns=colnames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The script"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# reset the dataframe\n",
    "\n",
    "# if there already is a file...\n",
    "if isfile(\"signs.pkl\"):\n",
    "    # ...load signs from that file\n",
    "    signs = pd.read_pickle(\"signs.pkl\")\n",
    "else:\n",
    "    # otherwise, set up an empty dataframe\n",
    "    signs = pd.DataFrame(columns=colnames)\n",
    "\n",
    "# show the number of signs in the dataframe\n",
    "print(len(signs))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# iterate over the list of types\n",
    "for sign_type in sign_types:\n",
    "            \n",
    "    # build the url\n",
    "    thepage = base_url + sign_type + '/'        \n",
    "\n",
    "    # call the url\n",
    "    stuff = requests.get(thepage)\n",
    "\n",
    "    # transform to soup using lxml parser\n",
    "    soup = bs4.BeautifulSoup(stuff.text, \"lxml\")\n",
    "\n",
    "    # extract the new signs from this page\n",
    "    new_signs = get_itemlist(soup)\n",
    "\n",
    "    # add the new signs to the dataframe\n",
    "    signs = signs.append(new_signs)\n",
    "\n",
    "    # print something to show how the process progresses\n",
    "    print(\"URL:\",thepage,flush=True)\n",
    "        \n",
    "                \n",
    "    # *** Tidy up the data and save to disk after each letter has been scraped ***\n",
    "        \n",
    "    # remove duplicates in case the same page has been scraped more than once\n",
    "    # signs = signs.drop_duplicates()\n",
    "        \n",
    "    # save the signs to a csv file after each page\n",
    "    signs.to_csv(\"signs.csv\")\n",
    "        \n",
    "    # save the signs to a pkl file after each page\n",
    "    signs.to_pickle(\"signs.pkl\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# how many signs are there in the dataframe?\n",
    "len(signs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# have a look at the first five items\n",
    "signs[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "signs[\"number_of_images\"] = signs[\"images\"].map(lambda x:len(x))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# histograms\n",
    "pp = signs.hist(figsize = (12,7))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# scatter diagram\n",
    "pp = signs[0:15].plot(kind='barh', use_index=\"True\")\n",
    "\n",
    "# look up syntax to use the sign_name column as the y-tick labels on the graph "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Statistical analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "signs.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That concludes our workshop today. \n",
    "\n",
    "Please come and talk to us at the conference if you have any questions, or contact us by email: Patrik Wikström (patrik.wikstrom@qut.edu.au) and Brenda Moon (brenda.moon@qut.edu.au).\n",
    "\n",
    "If you want to extend this example, you can find more information about the tools we have used on their websites, which are listed at the bottom of the [index page](index.ipnb)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}