GitHub - AmoebaFactor/SuperMario: A python module which support an other project named bububa.Lego provide several advance web scrape functions.

AmoebaFactor / SuperMario Public

forked from bububa/SuperMario

Notifications You must be signed in to change notification settings
Fork 0
Star 1

A python module which support an other project named bububa.Lego provide several advance web scrape functions.

syd.todayclose.com/

1 star 1 fork Branches Tags Activity

Star

Notifications

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 38 Commits
bububa		bububa
.gitignore		.gitignore
README.txt		README.txt
setup.py		setup.py

Repository files navigation

= About SuperMario =
    SuperMario is an advance web cralwer library written in python. It
provides a number of methods to mine data from kinds of sites.

== License ==
BSD License
See 'LICENSE' for details.

== Requirements ==
Platform: *nix like system (Unix, Linux, Mac OS X, etc.)
Python: 2.5+
Storage: mongodb
Some other python models:
    - simplejson
    - BeautifulSoup
    - eventlet 
    - PIL
    - pycurl
    - chardet
    - feedparser
    - mongokit
    - templatemaker
    - flickrapi
    - pyyaml
    - MySQLdb
    - dateutil

== Features ==
  + robots.txt protocol supported;
  + cache URL 's HTML;
  + normalize URL;
  + convert all content into unicode;
  + extract MainText from HTML by specific a * link-threshold *
  + convert partial RSS feed to full RSS feed;
  + proxies list support;
  + cookie keep support;
  + login support;