2020 08 20

Minutes Open Tech Call 2020-08-20

New participants

• @JacekTlaga who is exploring OCR-D for the Polish National Library
• @hnesk who's using OCR-D for DIY book scanning workflow
• @j23d who is implementing OCR-D in a web platform @StaatsbibliothekBerlin

New or upcoming features

• There was some disruption in ocrd_all because of side effects with introducing GNU parallel for its semaphore file locking into the Makefile. This has been fixed now
• We're on the final stretch to adapting all processors to the new one output file group for PAGE-XML and images convention. If you maintain an OCR-D processor project, please review pull requests we may have sent you.
• Implementation of the convention for storing file parameter files or bundling them with processors is now underway in OCR-D/core. If you have any comments or last-minute requirements, join the discussion.

API and OCR-D Butler

• @j23d explains the general architecture and purpose of OCR-D Butler
  • HTTP/REST API to processors and sequences of processors ("chains")
  • Based on Python
    • Flask for the web interface
    • restx to define the API
    • Swagger to present the API
    • Celery as a task manager
  • Processors are executed with OCR-D/core run_cli method
• OCR-D API should be showcased in a self-contained docker container for further discussion and easy testing
• Focus on a convention for translating CLI calls to HTTP calls, with inspiration from OCR-D Butler

Workflow format

• While individual processor steps can just be executed in sequence from a shell script, a proper workflow format has many advantages
  • Validation of input/output file groups
  • Ensure executables are installed
  • Validate parameters
  • Check quality of intermediate steps
  • Compose workflows from templates
  • ...
• OCR-D currently recommends two workflow engines:
  • ocrd process (part of OCR-D/core)
  • workflow-configuration (based on Makefiles, by @bertsky)
• We need an exchange format for workflows that should be (among other):
  • convertible to ocrd process, workflow-configuration Makefiles, OCR-D Butler chains...
  • simple to write manually
  • syntactically straightforward enough to support GUIs for editing them
  • able to express both simple static workflows yet be flexible enough to support upcoming quality assurance extensions
  • allows composition/inheritance
  • be parameterizable itself (i.e. should support variables and override mechanisms)
  • easy to integrate into the API
• We created a first draft of a spec for such a format, called OCRD-WF
  • Syntactically based on POSIX sh (i.e. shell script, a tiny subset of bash/zsh)
  • Supports variable assignments
    • But no agreement on the semantics and mechanics of variable assignments yet
  • Workflow steps are expressed in the same format as is used on the command-line
• Discussion
  • How important is dynamic processing?
    • If non-global scope is required (i.e. variables can change their value throughout the script), is shell script a good foundation?
    • Would it make sense to use a proper workflow configuration language like CWL?
    • Should quality assurance be a feature of the language, e.g. offering metrics in special variables?
    • Do we need real conditionals or just "good enough vs. not good enough" checks that could be implemented as processors?