Catch unhandled Browsershot exceptions in crawlFailed
#469
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
July 29, 2024 16:21
superpenguin612
changed the title
crawlFailed.crawlFailed
added 4 commits
July 29, 2024 17:42
|
Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #324.
I ran into this issue myself, and it ended up crashing the crawler prematurely. This PR catches any
ProcessFailedExceptions on the Browsershot method, and sends it to observers viacrawlFailed. I also included a test that loads a URL that never goes to network idle, triggering the 30s network timeout from Puppeteer. Originally, this creates an exception, but now it is caught appropriately.One note here is that exceptions like Puppeteer not finding a Chrome binary are no longer displayed to the user by default, they would need to see the exception in
crawlFailed. I'm not sure if there is a way to "fix" that, but that in theory should be what happens, anyway.Feel free to ask to use a different method of testing this; this test takes 30s to execute which may be undesirable (maybe use Browsershot's
waitForFunctionto get the 5s timeout, like what @kmcluckie posted)?