-
Notifications
You must be signed in to change notification settings - Fork 0
simple webcrawler using bs4 and requests.
License
amirongit/webider
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
webider is an absurd attempt to aggregate all the domains that can be found on the internet. - example proxy config ``` { "https": "socks5h://host:port", "http": "http://host:port" } ``` - how to ``` # after cloning the project, creating a virtual environment and setting up the db uri in config.json cd webider/src/ pip install -r ../requirements python main.py ``` - missing dependency for SOCKS? ``` pip install requests[socks] ```
About
simple webcrawler using bs4 and requests.