4.1.1

• Bump minimum required version of spatie/robots-txt to 1.0.1.

4.1.0

Respect robots.txt

4.0.5

• improved extensibility by removing php native type hinting of url, queue and crawler pool Closures

4.0.4 - do not follow links that have attribute `rel` set to `nofollow`

do not follow links that have attribute  set to

4.0.3

• Support both Illuminate's and Tighten's Collection.

4.0.2

• fix bugs when installing into a Laravel app

4.0.1

• fix bugs when installed in Laravel projects

4.0.0

• the CrawlObserver and CrawlProfile are upgraded from interfaces to abstract classes
• don't crawl tel: links

3.2.1

• fix endless loop

3.2.0

• add setCrawlObservers, addCrawlObserver