v2.54.0

[kba]

Added:

• METS Server: providing concurrent additional access to the METS file for parallel processing, METS Server #966
• Web API: Cache jobs in the processing server with an optional callback once processed, Processing server extension (#1046) #1069
• Web API: Lock pages output file groups of a workspace to prevent simultaneous non-additive access to workspaces, Processing server extension (#1046) #1069
• Web API: Support job dependency for caching complete fully-deterministic workflows, Processing server extension (#1046) #1069
• Web API: Processing server will start all ready requests, not just the first one, Processing server extension (#1046) #1069
• Web API: Workers will create on demand, retry attempts configurable via OCRD_NETWORK_WORKER_QUEUE_CONNECT_ATTEMPTS, Make ProcessingWorker create its queue by default #1093
• ocrd_utils.config to collect all configuration based on environment variables in one place, ocrd_utils.config to handle all environment variables #1081
• Processor CLI: Warn if a given page ID cannot be found in METS, Provide a warning if --page-id evaluates to an empty set #1088, raise ValueError if --page-id is provided but leads to empty result #1089

Changed:

• Processors now have worker and server subcommands, with separate --help, for starting processing worker/processor server, replace --agent-type with subcommand, #1032, #1085 #1087
• Move tf_disable_interactive_logs (to silence keras/tensorflow print statements) to ocrd_utils.logging and do not call on module-level, Can we properly solve the reason for tf1_disable_interactive_logs existence? #1090, move tf_disable_interactive_logs to utils/logging and remove calls fr… #1091
• 🔥 We do now properly distinguish between original and local-file FLocat, original URL will not be changed for downloads anymore, mets:file URL handling: keep remote links #323, Keep local copies of files in a separate mets:FLocat #1079
• 🔥 logging has been streamlined to be better usable as a library, Refactor logging #1080

---

This discussion was created from the release v2.54.0.

[kba]

Announcement in the OCR-D/Lobby:

METS Server

This solves the problem of concurrent access to the METS file by providing a server that handles all the METS operations so that an arbitrary number of processors can add files at the same time. With the METS server running, pages can now be processed in parallel, no need to split and merge workspaces anymore.

In practice, a METS server can be started with the ocrd workspace --mets-server-url URL server start command, where URL is either a TCP URL (http://host:port) or the path to a UNIX Domain socket:

ocrd workspace --mets-server-url /tmp/ws.sock server start

With the server running, provide --mets-server-url to processor calls to make use of it:

ocrd-tesserocr-recognize --mets-server-url /tmp/ws.sock ...

OcrdFile now have separate URL and filename fields.

Until now, when downloading files to the workspace, e.g. with ocrd workspace find --download, we took the URL of the mets:file, downloaded it to the workspace and then changed the reference to the local file. This was done to be compliant with the DFG Viewer profile but had the drawback, that the original URLs were not preserved and required post-processing to restore them.

With the newest release, we do not change the URL of mets:file anymore. Instead we add a second mets:FLocat to it, pointing to the local version. To remove those local filenames, you can use the ocrd workspace find --undo-download command.

For developers, this means that the distinction between OcrdFile.url and OcrdFile.local_filename is now meaningful and care must be taken to use the right one.

Web API getting more robust

We've been working on the implementation of the processing parts of the Web API in core. In the processing server, requests are now cached internally, with an optional callback when the jobs is finished. The processing server has a locking mechanism to prevent concurrent access - making full use of the METS server is in the works. Jobs in the processing server can now depend on, i.e. be blocked by, another job. This allows us to model a complete workflow as a set of jobs, also on page-level, that can be pushed as a whole into the processing server - a workflow endpoint that does that and supports the ocrd process syntax is scheduled for the next release.

We also changed the way processing workers and processor servers are started. We replaced the --agent-type option with subcommands worker and server for the processing worker and processor server respectively. Processing workers will automatically create RabbitMQ queues as necessary.

For example to start ocrd-tesserocr-recognize as a processing worker:

ocrd-tesserocr-recognize worker --queue ... --database ...

Cleaned up logging

We simplified our logging setup in ocrd_utils.logging to make it easier to use OCR-D/core as a library and not interfere with other logging conventions as much. With the newest version, it is no longer necessary to call initLogging unless you explicitly want the OCR-D logging conventions (e.g. for our command line tools and OCR-D processors). We do not change anything about the root logger anymore, we only add a handler to the ocrd logger. We also do not mess with the propagation of log messages in the log handler hierarchy anymore, instead relying on the standard behavior of python's logging.

In practice, this means that we strongly recommend you to use a logging namespace starting with ocrd. for all OCR-D-related logging. We will clean up the actual logging calls within our codebases in the next release

ocrd_utils.config

We've developed a config system for handling the environment variables we support in various places. This should be a transparent change for users, so no changes necessary but makes life easier for core developers, e.g. we generate the list of environment variables supported listed in ocrd --help with this.

Other changes

There were also some smaller bug fixes and additions, see the release notes for the details

We will release a new ocrd_all with updated core and some related processor updates soon, we will post here when that is available.

Thanks to @MehmedGIT, @joschrew, @tdoan2010, @bertsky and all other contributors!