Run the WebCrawler with the below command -
python3 scraper.py -u <url> --d <depth>
Example
python3 scraper.py -u "https://google.com" -d 2
-
Crawls recursively to a certain depth by user
-
Returns subdomains, links, javascript files for some recon
-
Flexible depth params for customizing the level or recursion