Access Website Stack improvement ideas

[anjackson]

To be broken down into issues and milestones...

• Push metrics from w3act_export to Prometheus.
• Resolve How should we handle playback of redirects to the web archive itself? webrecorder/pywb#591 (this is currently worked-around by dropping revisits)
• Resolve Problems playing back some Twitter records ukwa-pywb#61 (Fixing Twitter requires indexing POST requests etc.)
• Find any and all places where webrender-puppeteer is used with a proxy and ensure there is no trailing slash because this make puppeteer go crazy. Crash out if proxy setting has a trailing slash webrender-puppeteer#13
• Consider hooking Flask app into Sentry
• Make recently-crawled screenshots (and content!) available. To do so, we have to finish off the warc-server idea:
  • Update warc-server to maintain filename-HDFS mapping?
  • Then proxy across crawler warc-server and HDFS warc-server?
  • Or have a separate redirect service that bounces HDFS filenames to WebHDFS, and to the crawler warc-server otherwise?
  • Also, update warc-server to update shared file list in background thread. Could be separate services, using NGINX proxy_next_upstream to manage them.

See also ukwa/ukwa-access-api#1 and the other issues https://github.com/ukwa/ukwa-access-api/issues