Skip to content

univizor/scrapers

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

95 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

This repository is deprecated. Working on rewrite,... - Oto

Univizor Scrapers

List of all entities (Google Document)

Installation

How to run scraper?

Scraper Command
vsvo cd vsvo; ruby vsvo.rb > ./vsvo.log 2>&1
ung cd ung; ruby ung-library.rb > ./ung.log 2>&1
gea cd gea; ruby gea-college.rb > ./gea.log 2>&1

(*) gea: set REDOWNLOAD=true in gea-college.rb to fetch again

Database

Dumping latest structure on production.

mysqldump -d -h $DB_HOST -u $DB_USER -p$DB_PASS $DB_NAME \
--skip-comments --skip-add-drop-table > $DB_NAME.sql

Loading database localy.

mysql -h <host> -u <username> -p < univizor.sql

Python

mkvirtualenv --no-site-packages univizor
workon univizor
pip install --upgrade -r requirements.txt

Pipeline

Output

PDF file

About

[Deprecated] Scrapers written in hackathon.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published