You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've setup a crawler which works fine using spatie crawler on loads of different websites but on one website it returns a 403 which appears to be cloudflare blocking it. If I crawl this website through browsershot it crawls fine and gets past the cloudflare page, meaning it certainly isn't IP related. What could be causing this I'm not sure if it's something to do with the spatie crawler not having JS enabled but i've added the flag as shown in the image below. I noticed in the documentation you can add a setBrowserShot but not sure if this is something I need and what arguments I pass into if I do.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I've setup a crawler which works fine using spatie crawler on loads of different websites but on one website it returns a 403 which appears to be cloudflare blocking it. If I crawl this website through browsershot it crawls fine and gets past the cloudflare page, meaning it certainly isn't IP related. What could be causing this I'm not sure if it's something to do with the spatie crawler not having JS enabled but i've added the flag as shown in the image below. I noticed in the documentation you can add a setBrowserShot but not sure if this is something I need and what arguments I pass into if I do.
Beta Was this translation helpful? Give feedback.
All reactions