Webscraping NFL Player Data.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center> <h1>Prime DFS </h1></center>\n",
    "<center> <h1>Web Scraping NFL Data </h1></center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center><h1> Notebook Goal: </h1></center>\n",
    "<center><H4>Create a CSV file of relevant NFL stats for the 2020 Season</H4></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction to BeautifulSoup\n",
    "* A Web Scraping Library that organizes messy HTML and presents us with manageable Python objects\n",
    "* **Need to install the library before you can use it in Python**\n",
    "    * `pip install beautifulsoup4` - run this in your terminal\n",
    "* Full documentation available at https://www.crummy.com/software/BeautifulSoup/bs4/doc/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Importing the libraries\n",
    "* Urllib - Module allows us to read data from URLs\n",
    "    * Full urlopen documentation at https://docs.python.org/3/library/urllib.request.html\n",
    "* BeautifulSoup - Module allows us to transform the html into manageable Python objects\n",
    "    * Full documentation at https://www.crummy.com/software/BeautifulSoup/bs4/doc/\n",
    "* Pandas - Module allows access for easy to use data manipulation and analysis methods\n",
    "    * Full documentation at https://www.crummy.com/software/BeautifulSoup/bs4/doc/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from urllib.request import urlopen\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Creating the BeautifulSoup Object\n",
    "* Copy/Paste the url link to the football site we want to grab data from\n",
    "    * In this case we use string formatting to maninpulate the year\n",
    "    * Link in this use case can be found at https://www.pro-football-reference.com/years/2020/fantasy.htm\n",
    "* Assign the html variable as the result of placing our url link as the parameter to the urlopen function\n",
    "    * Will read and return the html data file\n",
    "* Assign the soup variable the result of placing our html variable as the parameter to the BeautifulSoupFunction\n",
    "    * This will be the main variable we will manipulate going forward\n",
    "    * The BeautifulSoup function transforms the html result into manageable python objects\n",
    "        * Try printing the soup variable below the next block to see what it returns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "year = 2020\n",
    "url = \"https://www.pro-football-reference.com/years/{}/fantasy.htm\".format(year) #string formatting to manipulate the url by a given year\n",
    "html = urlopen(url)\n",
    "soup = BeautifulSoup(html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Getting the Header Data for our CSV file\n",
    "* Use a list comprehension to find the text within the table header tag inside the second table row tag\n",
    "    * Use list slicing to remove the first column header\n",
    "    * Print the first five headers as proof"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Player', 'Tm', 'FantPos', 'Age', 'G']\n"
     ]
    }
   ],
   "source": [
    "headers = [th.getText() for th in soup.findAll('tr')[1].findAll('th')] #Find the second table row tag, find every table header column within it and extract the html text via the get_text method.\n",
    "headers = headers[1:] #Do not need the first (0 index) column header\n",
    "print(headers[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4. Getting the Table Row Data for our CSV file\n",
    "* Use a list comprehension to find the text within each table data cell within each table row\n",
    "    * Find all table rows not classed as thead\n",
    "        * Every 30 rows on this football-reference page contains table header rows\n",
    "    * Perform the list comprehension to grab player stats\n",
    "    * Remove the first two empty rows from the player stats list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "rows = soup.findAll('tr', class_ = lambda table_rows: table_rows != \"thead\") #Here we grab all rows that are not classed as table header rows - football reference throws in a table header row everyy 30 rows \n",
    "player_stats = [[td.getText() for td in rows[i].findAll('td')] # get the table data cell text from each table data cell\n",
    "                for i in range(len(rows))] #for each row\n",
    "player_stats = player_stats[2:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.5. Creating a Pandas DataFrame object\n",
    "* Simply place the player_stats list as the data parameter and the headers as the columns parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Player</th>\n",
       "      <th>Tm</th>\n",
       "      <th>FantPos</th>\n",
       "      <th>Age</th>\n",
       "      <th>G</th>\n",
       "      <th>GS</th>\n",
       "      <th>Cmp</th>\n",
       "      <th>Att</th>\n",
       "      <th>Yds</th>\n",
       "      <th>TD</th>\n",
       "      <th>...</th>\n",
       "      <th>TD</th>\n",
       "      <th>2PM</th>\n",
       "      <th>2PP</th>\n",
       "      <th>FantPt</th>\n",
       "      <th>PPR</th>\n",
       "      <th>DKPt</th>\n",
       "      <th>FDPt</th>\n",
       "      <th>VBD</th>\n",
       "      <th>PosRank</th>\n",
       "      <th>OvRank</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dalvin Cook</td>\n",
       "      <td>MIN</td>\n",
       "      <td>RB</td>\n",
       "      <td>25</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "      <td></td>\n",
       "      <td>223</td>\n",
       "      <td>251.5</td>\n",
       "      <td>260.5</td>\n",
       "      <td>237.0</td>\n",
       "      <td>133</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Derrick Henry</td>\n",
       "      <td>TEN</td>\n",
       "      <td>RB</td>\n",
       "      <td>26</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>12</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>207</td>\n",
       "      <td>221.0</td>\n",
       "      <td>224.0</td>\n",
       "      <td>214.0</td>\n",
       "      <td>118</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Tyreek Hill</td>\n",
       "      <td>KAN</td>\n",
       "      <td>WR</td>\n",
       "      <td>26</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>14</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>192</td>\n",
       "      <td>260.1</td>\n",
       "      <td>263.1</td>\n",
       "      <td>226.1</td>\n",
       "      <td>106</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Alvin Kamara</td>\n",
       "      <td>NOR</td>\n",
       "      <td>RB</td>\n",
       "      <td>25</td>\n",
       "      <td>11</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>12</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>195</td>\n",
       "      <td>263.1</td>\n",
       "      <td>269.1</td>\n",
       "      <td>229.1</td>\n",
       "      <td>106</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Kyler Murray</td>\n",
       "      <td>ARI</td>\n",
       "      <td>QB</td>\n",
       "      <td>23</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>264</td>\n",
       "      <td>387</td>\n",
       "      <td>2814</td>\n",
       "      <td>19</td>\n",
       "      <td>...</td>\n",
       "      <td>10</td>\n",
       "      <td></td>\n",
       "      <td></td>\n",
       "      <td>292</td>\n",
       "      <td>291.6</td>\n",
       "      <td>308.6</td>\n",
       "      <td>300.6</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Player   Tm FantPos Age   G  GS  Cmp  Att   Yds  TD  ...  TD 2PM  \\\n",
       "0     Dalvin Cook  MIN      RB  25  10  10    0    0     0   0  ...  14   3   \n",
       "1  Derrick Henry   TEN      RB  26  11  11    0    0     0   0  ...  12       \n",
       "2     Tyreek Hill  KAN      WR  26  11  11    0    0     0   0  ...  14       \n",
       "3    Alvin Kamara  NOR      RB  25  11   6    0    0     0   0  ...  12       \n",
       "4   Kyler Murray   ARI      QB  23  11  11  264  387  2814  19  ...  10       \n",
       "\n",
       "  2PP FantPt    PPR   DKPt   FDPt  VBD PosRank OvRank  \n",
       "0        223  251.5  260.5  237.0  133       1      1  \n",
       "1        207  221.0  224.0  214.0  118       2      2  \n",
       "2        192  260.1  263.1  226.1  106       1      3  \n",
       "3        195  263.1  269.1  229.1  106       3      4  \n",
       "4        292  291.6  308.6  300.6  100       1      5  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats = pd.DataFrame(player_stats, columns = headers)\n",
    "stats.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.7. Data Minipulation\n",
    "* Replace empty strings with zeroes, N/A values, column averages, etc.\n",
    "    * **It's your data you decide what works best for you**\n",
    "* Create a year column as the year variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Player</th>\n",
       "      <th>Tm</th>\n",
       "      <th>FantPos</th>\n",
       "      <th>Age</th>\n",
       "      <th>G</th>\n",
       "      <th>GS</th>\n",
       "      <th>Cmp</th>\n",
       "      <th>Att</th>\n",
       "      <th>Yds</th>\n",
       "      <th>TD</th>\n",
       "      <th>...</th>\n",
       "      <th>2PM</th>\n",
       "      <th>2PP</th>\n",
       "      <th>FantPt</th>\n",
       "      <th>PPR</th>\n",
       "      <th>DKPt</th>\n",
       "      <th>FDPt</th>\n",
       "      <th>VBD</th>\n",
       "      <th>PosRank</th>\n",
       "      <th>OvRank</th>\n",
       "      <th>Year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Dalvin Cook</td>\n",
       "      <td>MIN</td>\n",
       "      <td>RB</td>\n",
       "      <td>25</td>\n",
       "      <td>10</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>223</td>\n",
       "      <td>251.5</td>\n",
       "      <td>260.5</td>\n",
       "      <td>237.0</td>\n",
       "      <td>133</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Derrick Henry</td>\n",
       "      <td>TEN</td>\n",
       "      <td>RB</td>\n",
       "      <td>26</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>207</td>\n",
       "      <td>221.0</td>\n",
       "      <td>224.0</td>\n",
       "      <td>214.0</td>\n",
       "      <td>118</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Tyreek Hill</td>\n",
       "      <td>KAN</td>\n",
       "      <td>WR</td>\n",
       "      <td>26</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>192</td>\n",
       "      <td>260.1</td>\n",
       "      <td>263.1</td>\n",
       "      <td>226.1</td>\n",
       "      <td>106</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>2020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Alvin Kamara</td>\n",
       "      <td>NOR</td>\n",
       "      <td>RB</td>\n",
       "      <td>25</td>\n",
       "      <td>11</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>195</td>\n",
       "      <td>263.1</td>\n",
       "      <td>269.1</td>\n",
       "      <td>229.1</td>\n",
       "      <td>106</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>2020</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Kyler Murray</td>\n",
       "      <td>ARI</td>\n",
       "      <td>QB</td>\n",
       "      <td>23</td>\n",
       "      <td>11</td>\n",
       "      <td>11</td>\n",
       "      <td>264</td>\n",
       "      <td>387</td>\n",
       "      <td>2814</td>\n",
       "      <td>19</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>292</td>\n",
       "      <td>291.6</td>\n",
       "      <td>308.6</td>\n",
       "      <td>300.6</td>\n",
       "      <td>100</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>2020</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 33 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "           Player   Tm FantPos Age   G  GS  Cmp  Att   Yds  TD  ... 2PM 2PP  \\\n",
       "0     Dalvin Cook  MIN      RB  25  10  10    0    0     0   0  ...   3   0   \n",
       "1  Derrick Henry   TEN      RB  26  11  11    0    0     0   0  ...   0   0   \n",
       "2     Tyreek Hill  KAN      WR  26  11  11    0    0     0   0  ...   0   0   \n",
       "3    Alvin Kamara  NOR      RB  25  11   6    0    0     0   0  ...   0   0   \n",
       "4   Kyler Murray   ARI      QB  23  11  11  264  387  2814  19  ...   0   0   \n",
       "\n",
       "  FantPt    PPR   DKPt   FDPt  VBD PosRank OvRank  Year  \n",
       "0    223  251.5  260.5  237.0  133       1      1  2020  \n",
       "1    207  221.0  224.0  214.0  118       2      2  2020  \n",
       "2    192  260.1  263.1  226.1  106       1      3  2020  \n",
       "3    195  263.1  269.1  229.1  106       3      4  2020  \n",
       "4    292  291.6  308.6  300.6  100       1      5  2020  \n",
       "\n",
       "[5 rows x 33 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stats = stats.replace(r'', 0, regex=True) #replace the empty string\n",
    "stats['Year'] = year #create a new column as the year variable\n",
    "stats.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.8. Creating the CSV file\n",
    "* Enter your unique file path as the parameter string to the to_csv method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "stats.to_csv('2020playerstats.csv') #add your desired path prior within the string here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Putting it all together\n",
    "* Creating a function that creates a CSV file of player data\n",
    "* Only parameter is the year of data you wish to create data from"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def player_csv(year):\n",
    "\n",
    "    url = \"https://www.pro-football-reference.com/years/{}/fantasy.htm\".format(year)\n",
    "    html = urlopen(url)\n",
    "    soup = BeautifulSoup(html)\n",
    "\n",
    "    headers = [th.getText() for th in soup.findAll('tr')[1].findAll('th')] #Find the second table row tag, find every table header column within it and extract the html text via the get_text method.\n",
    "    headers = headers[1:] #Do not need the first (0 index) column header\n",
    "    \n",
    "    rows = soup.findAll('tr', class_ = lambda table_rows: table_rows != \"thead\") #Here we grab all rows that are not classed as table header rows - football reference throws in a table header row everyy 30 rows \n",
    "    player_stats = [[td.getText() for td in rows[i].findAll('td')] #get the table data cell text from each table data cell\n",
    "                    for i in range(len(rows))] #for each row\n",
    "    player_stats = player_stats[2:]\n",
    "\n",
    "    stats = pd.DataFrame(player_stats, columns = headers)\n",
    "    \n",
    "    stats = stats.replace(r'', 'N/A', regex=True)\n",
    "    stats['Year'] = year\n",
    "    \n",
    "    stats.to_csv('{}playerstats.csv'.format(year)) #add your desired path to the function\n",
    "    \n",
    "    print(\"Player data for the year {} has been created.\".format(year))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "player_csv(2019)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}