Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AllUrlsValid validator should use grequests to parallelise checks #125

Open
jace opened this issue Jun 27, 2016 · 0 comments
Open

AllUrlsValid validator should use grequests to parallelise checks #125

jace opened this issue Jun 27, 2016 · 0 comments

Comments

@jace
Copy link
Member

jace commented Jun 27, 2016

Rather than checking one URL at a time, as at present, which blocks on I/O, the AllUrlsValid validator should use grequests to parallelise the requests and check one as it returns.

The validator also checks the same URL twice if it appears twice, depending on the cache to prevent a second lookup. Since the cache isn't guaranteed when checking requests in parallel (and as an externally hosted cache is also a network call away), AllUrlsValid should also de-dupe URLs before doing the checks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant