1.0.3
- Request rearchival by default, add server-specific setting.
Background and Intent
Archive.org rate-limits pretty heavily when looking up previously-archived pages, but does not seem to rate-limit nearly as badly when requesting a page be rearchived immediately.
Previously, logic was written so lookups were always done by default and it was configurable whether or not rearchival was requested. This was done with the intention of lessening the load on Archive.org, but the result was Archive.org always responded with 429 Too Many Requests
. Now lookups are skipped by default and rearchival is requested immediately.
The result should be longer wait times for the archive URL but the bot should return a URL on the first attempt much more often, and the URL it returns will be as captured when the bot was first called.