Sample web scrape (simple).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# This is a sample of a web scrape\n",
    "\n",
    "#### Collecting data from a webpage\n",
    "\n",
    "This page contains a demonstration of grabbing data from a webpage and turning it into a dataframe for further analysis. The additional analysis is not conducted here, the focus is to demonstrate a web scraping program to collect data from an internet page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Importing libraries\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "from lxml import html\n",
    "import requests\n",
    "import urllib\n",
    "\n",
    "import pandas as pd\n",
    "from pandas import Series, DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Retreiving data from the web page\n",
    "\n",
    "page = requests.get('http://econpy.pythonanywhere.com/ex/001.html')\n",
    "tree = html.fromstring(page.content)\n",
    "url = ('http://econpy.pythonanywhere.com/ex/001.html')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\n<html>\\n<head>\\n    <meta charset=\"utf-8\">\\n    <title>Items 1 to 20 -- Example Page 1</title>\\n    <script type=\"text/javascript\">\\n      var _gaq = _gaq || [];\\n      _gaq.push([\\'_setAccount\\', \\'UA-23648880-1\\']);\\n      _gaq.push([\\'_trackPageview\\']);\\n      _gaq.push([\\'_setDomainName\\', \\'econpy.org\\']);\\n    </script>\\n</head>\\n<body>\\n<div align=\"center\">1, <a href=\"http://econpy.pythonanywhere.com/ex/002.html\">[<font color=\"green\">2</font>]</a>, <a href=\"http://econpy.pythonanywhere.com/ex/003.html\">[<font color=\"green\">3</font>]</a>, <a href=\"http://econpy.pythonanywhere.com/ex/004.html\">[<font color=\"green\">4</font>]</a>, <a href=\"http://econpy.pythonanywhere.com/ex/005.html\">[<font color=\"green\">5</font>]</a></div>\\n<div title=\"buyer-info\">\\n  <div title=\"buyer-name\">Carson Busses</div>\\n  <span class=\"item-price\">$29.95</span><br>\\n</div>\\n<div title=\"buyer-info\">\\n  <div title=\"buyer-name\">Earl E. Byrd</div>\\n  <span class=\"item-price\">$8.37</span><br>\\n</div>\\n<div title=\"buyer-info\"'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is a look of the code behind the web page. We need to know\n",
    "# what the element tags are to grab the correct information.\n",
    "\n",
    "urllib.urlopen(url).read(1000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### The information we want are the buyer and prices.\n",
    "\n",
    "##### These are under \"div title= \"buyer-name\" and\n",
    "##### \"span class= \"item-price\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# This will create a list of buyers:\n",
    "buyers = tree.xpath('//div[@title= \"buyer-name\"]/text()')\n",
    "\n",
    "# This will create a list of prices\n",
    "prices = tree.xpath('//span[@class= \"item-price\"]/text()')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Carson Busses', 'Earl E. Byrd', 'Patty Cakes', 'Derri Anne Connecticut', 'Moe Dess', 'Leda Doggslife', 'Dan Druff', 'Al Fresco', 'Ido Hoe', 'Howie Kisses', 'Len Lease', 'Phil Meup', 'Ira Pent', 'Ben D. Rules', 'Ave Sectomy', 'Gary Shattire', 'Bobbi Soks', 'Sheila Takya', 'Rose Tattoo', 'Moe Tell']\n"
     ]
    }
   ],
   "source": [
    "# Let's see the buyers\n",
    "\n",
    "print buyers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['$29.95', '$8.37', '$15.26', '$19.25', '$19.25', '$13.99', '$31.57', '$8.49', '$14.47', '$15.86', '$11.11', '$15.98', '$16.27', '$7.50', '$50.85', '$14.26', '$5.68', '$15.00', '$114.07', '$10.09']\n"
     ]
    }
   ],
   "source": [
    "# Let's see the prices\n",
    "\n",
    "print prices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###### Since the web scraper grabbed the data systemmatically, we know  the buyer and price matches up in the separate lists.\n",
    "###### Knowing that, we can combine it into a data frame so it's easier to view and for later analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Combining the separate lists into a single dataframe\n",
    "\n",
    "buyer_price_df = DataFrame(zip(buyers,prices), \n",
    "                           columns = (\"Buyer\", \"Price\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having the data in a dataframe is nice to work with in Python. However, it doesn't help those who don't know python!\n",
    "So everybody can access the data and work with it, we can export it to a .csv file so others can view it in Excel or another\n",
    "spreadsheet program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Exporting dataframe to .csv file\n",
    "\n",
    "buyer_price_df.to_csv('buyer_price_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}