Skip to content

Latest commit

 

History

History

csv

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
ncaa_games_2002.csv
ncaa_games_2002.csv
 
 
ncaa_games_2003.csv
ncaa_games_2003.csv
 
 
ncaa_games_2004.csv
ncaa_games_2004.csv
 
 
ncaa_games_2005.csv
ncaa_games_2005.csv
 
 
ncaa_games_2006.csv
ncaa_games_2006.csv
 
 
ncaa_games_2007.csv
ncaa_games_2007.csv
 
 
ncaa_games_2008.csv
ncaa_games_2008.csv
 
 
ncaa_games_2009.csv
ncaa_games_2009.csv
 
 
ncaa_games_2010.csv
ncaa_games_2010.csv
 
 
ncaa_games_2011.csv
ncaa_games_2011.csv
 
 
ncaa_games_2012.csv
ncaa_games_2012.csv
 
 
ncaa_games_2013.csv
ncaa_games_2013.csv
 
 
ncaa_games_2014.csv
ncaa_games_2014.csv
 
 
ncaa_games_2015.csv
ncaa_games_2015.csv
 
 
ncaa_games_2016.csv
ncaa_games_2016.csv
 
 
ncaa_games_2017.csv
ncaa_games_2017.csv
 
 
ncaa_players_2002.csv
ncaa_players_2002.csv
 
 
ncaa_players_2003.csv
ncaa_players_2003.csv
 
 
ncaa_players_2004.csv
ncaa_players_2004.csv
 
 
ncaa_players_2005.csv
ncaa_players_2005.csv
 
 
ncaa_players_2006.csv
ncaa_players_2006.csv
 
 
ncaa_players_2007.csv
ncaa_players_2007.csv
 
 
ncaa_players_2008.csv
ncaa_players_2008.csv
 
 
ncaa_players_2009.csv
ncaa_players_2009.csv
 
 
ncaa_players_2010.csv
ncaa_players_2010.csv
 
 
ncaa_players_2011.csv
ncaa_players_2011.csv
 
 
ncaa_players_2012.csv
ncaa_players_2012.csv
 
 
ncaa_players_2013.csv
ncaa_players_2013.csv
 
 
ncaa_players_2014.csv
ncaa_players_2014.csv
 
 
ncaa_players_2015.csv
ncaa_players_2015.csv
 
 
ncaa_players_2016.csv
ncaa_players_2016.csv
 
 
ncaa_players_2017.csv
ncaa_players_2017.csv
 
 
ncaa_schools.csv
ncaa_schools.csv
 
 

README.md

This data is scraped from NCAA's stats tool on various pages.

Beware that random columns are null, which is indicated by - in the HTML. Sometimes there is no data for a team, but there will be a blank row instead of no rows.

Only make one request at a time or NCAA will temporarily block you.

fetch_csvs.py

See fetch_csvs.py to do this automatically. Run with --help for additional arguments like the years to download.

We also do some cleanup:

  • Replace '-' with empty string
  • Remove blank rows
  • Convert height from feet-inches to just inches
  • Remove commas in attendence numbers

NCAA API Info

School Info

  • Command: ./fetch_csvs.py get_schools
  • URL: http://web1.ncaa.org/stats/StatsSrv/careersearch
  • How: Just navigate to that page normally.
  • Elements: The "School" dropdown contains names as text and ID's as values. Note that the first <option> is "All", so make sure to skip that.
  • XPath: //select[@name='searchOrg']/option[position()>1]

Player Info

  • Command: ./fetch_csvs.py get_players
  • URL: http://web1.ncaa.org/stats/StatsSrv/careerteam
  • How: From the search page in School Info, select your school, year, and sport, click "Search", then click on the school's name in the bottom table.
    • As a script, POST this data to the URL:
      • sortOn: 0
      • doWhat: "display"
      • playerId: -100
      • coachId: -100
      • orgId: The school's ID
      • academicYear: The year (for 2015-16, send "2016")
      • division: 1
      • sportCode: "MBB"
      • idx: ""
  • Elements: The table of player statistics at the bottom. Note that we can get the player ID from the link href in the first column. Make sure to skip the headers.
  • XPath: //table[@class='statstable'][2]//tr[position()>3]

Game Info

  • Command: ./fetch_csvs.py get_games
  • URL: http://web1.ncaa.org/stats/exec/records
  • How: From the Player Info page, click "View [Team Name] Year-By-Year W/L Record", then click a row in the "W-L" column.
    • As a script, POST this data to the URL:
      • academicYear: The year (for 2015-16, send "2016")
      • orgId: The school ID
      • sportCode: "MBB"
  • Elements: The second table. Note that we can get the opponent's ID from the link href in the first column.
  • XPath: //form[@name='orgRecords']/table[2]/tr[position()>1]