Add option to honor robots.txt information

[paceaux]

Provide an option to honor robots.txt file.

1. It looks for robots.txt to begin with
2. I will find the sitemap from there if it's listed
3. It will ignore any pages explicitely listed in robots.txt

[paceaux]

Spec for robots is here

Things to note:

• a page can have a meta tag <meta name="robots">
• a robots file can have a disallow
• Pages or directories can be disallowed

[paceaux]

Honoring robots will mean

1. first grab a robots.txt
2. Collect the disallow rules from it
3. Generate sitemap based on allowed pages
4. When scanning a page, first checking if a <meta name=robots> exists, and then honoring it

[paceaux]

Question:

Where to dump robots info. If anywhere

A sitemap could have a ton of links, or crawling could.
if a user sees pages missing, they'll want to know why. If we add robots info into the sitemap json, could ... confuse stuff?

not everyone looks in logs though

[paceaux]

also, where do we decide to exclude a page

setSitemaps() would be a great place because that takes the sitemap file and does an addLinks() method, and we could just filter out the pages that are matches.

but then it's not immediately obvious that happened.

We also want to truly honor a robots file and not go to the page if we have it in the robots file.

[paceaux]

Robots class is created.

we will have a -b option to honor robots.
This will only affect crawling. But it will produce a disallowed.json file.

The next step will be to also read a disallowed.json file so that now this covers both robots behavior and "filtering" of urls. this should probably be covered in the "filtering urls" issue.

[paceaux]

This is mostly finished I think, but this needs to be tested on a live site.

[paceaux]

Finally found a better explanation of syntax: https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt#syntax