UKWA PYWB Access System

This repository represents a customized extension of pywb for use with the UK Web Archive. It includes custom features:

• Custom UI for UKWA
  • text selection limits
  • transclusions support
• Single-Concurrent Lock System
• SELECT_WORD_LIMIT
• Video Support/hacks TBC
• content_type_redirects ? TBC
• download blocks ? TBC
• Memento Prefer Header initial implementation (See this issue for further discussion) (TBC: now part of PyWB) ???
• ePub/PDF reader support TBA
• NPLD Viewer Access Control Setup

...and configuration files for standard PyWB features:

• Localization
• Access Controls
• HTTP/S Proxy Mode
• NPLD Viewer Access Control Setup
• Memento Prefer Header initial implementation (See this issue for further discussion)

This repository builds an ukwa-pywb container image, which extends the offical pywb container image release.

In the past, additional PyWB functionality has been developed on the ukwa/pywb fork of pywb, but this has now all been merged upstream. This allows us to depend on PyWB directly rather than our own release.

Upgrading PyWB

This project can be upgraded by simply the version of the PyWB docker container referred to in the Dockerfile. In general, upgrades should usually be smooth, but as in this case, sometimes changes have been made to PyWB that affects the things we've modified or extended, like the banner template handling.

To check this, the new image can be built and tested using the integration test system, as outlined below.

Once tested, the version can be tagged, using the PyWB version as a base. i.e. the version of ukwa/ukwa-pywb based on webrecorder/pywb:2.6.2 should be 2.6.2. If any further releases are required to resolve unexpected problems, while sticking to the same version of PyWB, a point suffix can be added, e.g. 2.6.2.1, 2.6.2.2 and so on.

Once the image has been built, it should be rolled out across the relevant ukwa-services. This includes the website, w3act and reading room service stacks.

Development Setup

To build and run from a checked-out repository, you can set up a virtualenv (or similar) using Python 3.7 or later. Install the dependencies e.g.

python setup.py install

Then run

uwsgi uwsgi.ini

And ukwa-pywb should start using the configuration in uwsgi.ini, i.e. available on port 8080, and using the CDX and WARC files in ./integration-test/test-data/. If the port is in use, create a copy of the uwsgi.ini file configuring alternative port(s) and use that instead.

To perform more complex testing that includes additional services (like Redis), you can use the integration test setup. First

cd integration-test/

Then you can use this to re-build the containerized version locally:

docker-compose build pywb

And run it using:

docker-compose up populate pywb

The first time you do this, the populate container will make the test data available and populate the system with it. You will need to do this again if you fully remove the integration testing containers. However, if you're just re-building pywb for testings, you can just do this:

docker-compose up pywb

When running locally, the service should be available at: http://localhost:8081 and should contain:

• A copy of the The Archival Acid Test: http://localhost:8081/open-access/*/http://acid.matkelly.com/
• Example HTTP 451 resource: http://localhost:8081/open-access/20180203004147/http://www.cs.odu.edu/~mkelly/acid/externalScript.js

See Integration Tests for more details.

Deployment and Configuration

The project can be deployed locally or in Docker.

• See Deployment for more information about deploying this repository and running tests.

• See Configuration for more information about the config.yaml file and its options.

• See Integration Tests on how to run the included integration test suite.