You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Rather than checking one URL at a time, as at present, which blocks on I/O, the AllUrlsValid validator should use grequests to parallelise the requests and check one as it returns.
The validator also checks the same URL twice if it appears twice, depending on the cache to prevent a second lookup. Since the cache isn't guaranteed when checking requests in parallel (and as an externally hosted cache is also a network call away), AllUrlsValid should also de-dupe URLs before doing the checks.
The text was updated successfully, but these errors were encountered:
Rather than checking one URL at a time, as at present, which blocks on I/O, the
AllUrlsValid
validator should use grequests to parallelise the requests and check one as it returns.The validator also checks the same URL twice if it appears twice, depending on the cache to prevent a second lookup. Since the cache isn't guaranteed when checking requests in parallel (and as an externally hosted cache is also a network call away),
AllUrlsValid
should also de-dupe URLs before doing the checks.The text was updated successfully, but these errors were encountered: