Breaking change: moving shouldCrawl from CrawProfile to crawl observers

[Velka-DEV]

Hello everyone!

As the title suggests, I'd like to propose a significant change regarding the pre-crawling filter behavior.

Currently, we're limited to using only one CrawlProfile for the crawler. This means all checks need to be bundled into a single, general class.

A more effective approach would be to abstract the shouldCrawl method into the CrawlObserver, allowing each observer to implement its own logic.

Then, we would check if ANY observer wishes to initiate the crawl. If at least one does, the page gets crawled and the information is sent only to the relevant observers, not all. I believe this would be a much more efficient and targeted behavior.

What are your thoughts on this?

Happy New Year to all of you 🎉 !

[Velka-DEV]

Hey @freekmurze, sorry for the ping.

I just would like to have your approval before working on this breaking change.

Also I think that it would be possible to keep the CrawlProfile to not introduce a breaking change and use it as a separated predicate more global for the crawler but that seem a bit a duplicated code to me.

What do you think about it ?

[freekmurze]

I'm not entirely sure if I follow why this would be beneficial, but feel free to convince me with code. I highly prefer a non-breaking version of this so we don't have to tag a new major version should the PR be accepted.