Skip to content

simple webcrawler using bs4 and requests.

License

Notifications You must be signed in to change notification settings

amirongit/webider

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

webider is an absurd attempt to aggregate all the domains that can be found on the internet.

- example proxy config
```
{
    "https": "socks5h://host:port",
    "http": "http://host:port"
}
```

- how to
```
# after cloning the project, creating a virtual environment and setting up the db uri in config.json
cd webider/src/
pip install -r ../requirements
python main.py
```

- missing dependency for SOCKS?
```
pip install requests[socks]
```