diff --git a/README.md b/README.md index 2fa4f131..124e55f9 100644 --- a/README.md +++ b/README.md @@ -286,6 +286,9 @@ Options: telemetry purposes. Defaults to "{}" [string] [default: "{}"] + -r, --followRobots Option for crawler to adhere to robots.txt + rules if it exists + [string] [choices: "yes", "no"] [default: "no"] Examples: To scan sitemap of website:', 'node cli.js -c [ 1 | Sitemap ] -d -u -w