New processor API

.pylintrc

@@ -1,19 +1,22 @@
 [MASTER]
-extension-pkg-whitelist=lxml
-ignored-modules=cv2,tesserocr,ocrd.model
+extension-pkg-whitelist=lxml,pydantic
+ignored-modules=cv2,tesserocr,ocrd_models.ocrd_page_generateds
+ignore-paths=ocrd_page_generateds.py
+ignore-patterns=.*generateds.*
 
 [MESSAGES CONTROL]
-ignore-patterns='.*generateds.*'
 disable =
     fixme,
-    E501,
+    line-too-long,
+    consider-using-f-string,
+    logging-fstring-interpolation,
     trailing-whitespace,
     logging-not-lazy,
     inconsistent-return-statements,
+    disallowed-name,
     invalid-name,
     line-too-long,
     missing-docstring,
-    no-self-use,
     wrong-import-order,
     too-many-nested-blocks,
     superfluous-parens,
@@ -25,13 +28,9 @@ disable =
     ungrouped-imports,
     useless-object-inheritance,
     useless-import-alias,
-    bad-continuation,
     no-else-return,
     logging-not-lazy
 
-[FORMAT]
-no-space-check=empty-line
-
 [DESIGN]
 # Maximum number of arguments for function / method
 max-args=12
@@ -40,7 +39,7 @@ max-locals=30
 # Maximum number of return / yield for function / method body
 max-returns=12
 # Maximum number of branch for function / method body
-max-branchs=30
+max-branches=30
 # Maximum number of statements in function / method body
 max-statements=60
 # Maximum number of parents for a class (see R0901).

CHANGELOG.md

@@ -5,6 +5,164 @@ Versioned according to [Semantic Versioning](http://semver.org/).
 
 ## Unreleased
 
+## [3.0.0b7] - 2024-11-12
+
+Fixed:
+ - `initLogging`: only add root handler instead of multiple redundant handlers with `propagate=false`
+ - `setOverrideLogLevel`: override all currently active loggers' level
+
+Changed:
+ - :fire: logging: increase default root (not `ocrd`) level from `INFO` to `WARNING`
+ - :fire: `initLogging`: do not remove any previous handlers/levels, unless `force_reinit`
+ - :fire: `disableLogging`: remove all handlers, reset all levels - instead of being selective
+ - :fire: Processor: replace `weakref` with `__del__` to trigger `shutdown`
+ - :fire: `OCRD_MAX_PARALLEL_PAGES>1`: log via `QueueHandler` in subprocess, `QueueListener` in main
+
+## [3.0.0b6] - 2024-10-30
+
+Fixed:
+ - `OcrdMets.get_physical_pages`: cover `return_divs` w/o `for_fileIds` and `for_pageIds`
+
+Changed:
+ - :fire: `ocrd_utils.initLogging`: also add handler to root logger (as in file config),
+   but disable message propagation to avoid duplication
+ - only import `ocrd_network` in `src/ocrd/decorators/__init__.py` once needed
+ - `Processor.process_page_file`: skip computing `process_page_pcgts` if output already exists,  
+   but `OCRD_EXISTING_OUTPUT!=OVERWRITE`
+ - :fire: `OCRD_MAX_PARALLEL_PAGES>1`: switch from multithreading to multiprocessing, depend on
+   `loky` instead of stdlib `concurrent.futures`
+ - `OCRD_PROCESSING_PAGE_TIMEOUT>0`: actually enforce timeout within worker
+ - `OCRD_MAX_MISSING_OUTPUTS>0`: abort early if too many failures already, prospectively
+ - `Processor.process_workspace`: split up into overridable sub-methods:
+   - `process_workspace_submit_tasks` (iterate input file group and schedule page tasks)
+     - `process_workspace_submit_page_task` (download input files and submit single page task)
+   - `process_workspace_handle_tasks` (monitor page tasks and aggregate results)
+     - `process_workspace_handle_page_task` (await single page task and handle errors)
+
+
+## [3.0.0b5] - 2024-09-16
+
+Fixed:
+  - tests: ensure `ocrd_utils.config` gets reset whenever changing it globally
+  - `OcrdMetsServer.add_file`: pass on `force` kwarg
+  - `ocrd.cli.workspace`: consistently pass on `--mets-server-url` and `--backup`
+  - `ocrd.cli.validate "tasks"`: pass on `--mets-server-url`
+  - `ocrd.cli.bashlib "input-files"`: pass on `--mets-server-url`
+  - `lib.bash input-files`: pass on `--mets-server-url`, `--overwrite`, and parameters
+  - `lib.bash`: fix `errexit` handling
+  - `ocrd.cli.ocrd-tool "resolve-resource"`: forgot to actually print result
+
+Changed:
+  - :fire: `Processor` / `Workspace.add_file`: always `force` if `OCRD_EXISTING_OUTPUT==OVERWRITE`
+  - :fire: `Processor.verify`: revert 3.0.0b1 enforcing cardinality checks (stay backwards compatible)
+  - :fire: `Processor.verify`: check output fileGrps, too
+     (must not exist unless `OCRD_EXISTING_OUTPUT=OVERWRITE|SKIP` or disjoint `--page-id` range)
+  - lib.bash `input-files`: do not try to validate tasks here (now covered by `Processor.verify()`)
+  - `run_processor`: be robust if `ocrd_tool` is missing `steps`
+  - `PcGtsType.PageType.id` via `make_xml_id`: replace `/` with `_`
+
+Added:
+  - `OcrdPage`: new `PageType.get_ReadingOrderGroups()` to retrieve recursive RO as dict
+  - ocrd.cli.workspace `server`: add subcommands `reload` and `save`
+  - METS Server: export and delegate `physical_pages`
+  - processor CLI: delegate `--resolve-resource`, too
+  - `Processor.process_page_file` / `OcrdPageResultImage`: allow `None` besides `AlternativeImageType`
+
+## [3.0.0b4] - 2024-09-02
+
+Fixed:
+
+  * `Processor.metadata_location`: `src` workaround respects namespace packages, qurator-spk/eynollah#134
+  * `Workspace.reload_mets`: handle ClientSideOcrdMets as well
+
+## [3.0.0b3] - 2024-08-30
+
+Added:
+
+  * `OcrdConfig.reset_defaults` to reset config variables to their defaults
+
+## [3.0.0b2] - 2024-08-30
+
+Added:
+ - `Processor.max_workers`: class attribute to control per-page parallelism of this implementation
+ - `Processor.max_page_seconds`: class attribute to control per-page timeout of this implementation
+ - `OCRD_MAX_PARALLEL_PAGES` for whether and how many workers should process pages in parallel
+ - `OCRD_PROCESSING_PAGE_TIMEOUT` for whether and how long processors should wait for single pages
+ - `OCRD_MAX_MISSING_OUTPUTS` for maximum rate (fraction) of pages before making `OCRD_MISSING_OUTPUT=abort`
+
+Fixed:
+  - `disableLogging`: also re-instate root logger to Python defaults
+
+## [3.0.0b1] - 2024-08-26
+
+Fixed:
+  - actually apply CLI `--log-filename`, and show in `--help`
+  - adapt to Pillow changes
+  - `ocrd workspace clone`: do pass on `--file-grp` (for download filtering)
+
+Changed:
+  - :fire: `ocrd_utils`, `ocrd_models`, `ocrd_modelfactory`, `ocrd_validators` and `ocrd_network` are not published
+    as separate packages anymore, everything is contained in `ocrd` - you should adapt your `requirements.txt` accordingly
+  - :fire: `Processor.parameter` now a property (attribute always exists, but `None` for non-processing contexts)
+  - :fire: `Processor.parameter` is now a `frozendict` (contents immutable)
+  - :fire: `Processor.parameter` validate when(ever) set instead of (just) the constructor
+  - setting `Processor.parameter` will also trigger (`Processor.shutdown() and) `Processor.setup()`
+  - `get_processor(... instance_caching=True)`: use `min(max_instances, OCRD_MAX_PROCESSOR_CACHE)`
+  - :fire: `Processor.verify` always validates fileGrp cardinalities (because we have `ocrd-tool.json` defaults now)
+  - :fire: `OcrdMets.add_agent` without positional arguments
+  - `ocrd bashlib input-files` now uses normal Processor decorator, and gets passed actual `ocrd-tool.json` and tool name
+    from bashlib's `ocrd__wrap`
+
+Added:
+  - `Processor.metadata_filename`: expose to make local path of `ocrd-tool.json` in Python distribution reusable+overridable
+  - `Processor.metadata_location`: expose to make absolute path of `ocrd-tool.json` reusable+overridable
+  - `Processor.metadata_rawdict`: expose to make in-memory contents of `ocrd-tool.json` reusable+overridable
+  - `Processor.metadata`: expose to make validated and default-expanded contents of `ocrd-tool.json` reusable+overridable
+  - `Processor.shutdown`: to shut down processor after processing, optional
+  - `Processor.max_instances`: class attribute to control instance caching of this implementation
+
+## [3.0.0a2] - 2024-08-22
+
+Changed:
+ - :fire: `OcrdPage` as proxy of `PcGtsType` instead of alias; also contains `etree` and `mapping` now
+ - :fire: `page_from_file`: removed kwarg `with_tree` - use `OcrdPage.etree` and `OcrdPage.mapping` instead
+ - :fire: `Processor.zip_input_files` now can throw `ocrd.NonUniqueInputFile` and `ocrd.MissingInputFile`
+   (the latter only if `OCRD_MISSING_INPUT=ABORT`)
+ - :fire: `Processor.zip_input_files` does not by default use `require_first` anymore
+   (so the first file in any input file tuple per page can be `None` as well)
+ - :fire: no more `Workspace.overwrite_mode`, merely delegate to `OCRD_EXISTING_OUTPUT=OVERWRITE`
+ - :art: improve on docs result for `ocrd_utils.config`
+
+Added:
+  - :point_right: `OCRD_DOWNLOAD_INPUT` for whether input files should be downloaded before processing
+  - :point_right: `OCRD_MISSING_INPUT` for how to handle missing input files (**`SKIP`** or `ABORT`)
+  - :point_right: `OCRD_MISSING_OUTPUT` for how to handle processing failures (**`SKIP`** or `ABORT` or `COPY`)
+     the latter behaves like ocrd-dummy for the failed page(s)
+  - :point_right: `OCRD_EXISTING_OUTPUT` for how to handle existing output files (**`SKIP`** or `ABORT` or `OVERWRITE`)
+  - new CLI option `--debug` as short-hand for `ABORT` choices above
+  - `Processor.logger` set up by constructor already (for re-use by processor implementors)
+  - `default`-expand and validate `ocrd_tool.json` in `Processor` constructor, log invalidities
+  - handle JSON `deprecation` in `ocrd_tool.json` by reporting warnings
+
+## [3.0.0a1] - 2024-08-15
+
+Changed:
+  - :fire: Deprecate `Processor.process`
+  - update spec to v3.25.0, which requires annotating fileGrp cardinality in `ocrd-tool.json`
+  - :fire: Remove passing non-processing kwargs to `Processor` constructor, add as members  
+     (i.e. `show_help`, `dump_json`, `dump_module_dir`, `list_resources`, `show_resource`, `resolve_resource`)
+  - :fire: Deprecate passing processing arg / kwargs to `Processor` constructor  
+     (i.e. `workspace`, `page_id`, `input_file_grp`, `output_file_grp`; now all set by `run_processor`)
+  - :fire: Deprecate passing `ocrd-tool.json` metadata to `Processor` constructor
+  - `ocrd.processor`: Handle loading of bundled `ocrd-tool.json` generically
+
+Added:
+  - `Processor.process_workspace`: process a complete workspace, with default implementation
+  - `Processor.process_page_file`: process an OcrdFile, with default implementation
+  - `Processor.process_page_pcgts`: process a single OcrdPage, produce a single OcrdPage, required to implement
+  - `Processor.verify`: handle fileGrp cardinality verification, with default implementation
+  - `Processor.setup`: to set up processor before processing, optional
+
 ## [2.68.0] - 2024-08-23
 
 Changed:
@@ -2164,6 +2322,14 @@ Fixed
 Initial Release
 
 <!-- link-labels -->
+[3.0.0b6]: ../../compare/v3.0.0b6..v3.0.0b5
+[3.0.0b5]: ../../compare/v3.0.0b5..v3.0.0b4
+[3.0.0b4]: ../../compare/v3.0.0b4..v3.0.0b3
+[3.0.0b3]: ../../compare/v3.0.0b3..v3.0.0b2
+[3.0.0b2]: ../../compare/v3.0.0b2..v3.0.0b1
+[3.0.0b1]: ../../compare/v3.0.0b1..v3.0.0a2
+[3.0.0a2]: ../../compare/v3.0.0a2..v3.0.0a1
+[3.0.0a1]: ../../compare/v3.0.0a1..v2.67.2
 [2.68.0]: ../../compare/v2.68.0..v2.67.2
 [2.67.2]: ../../compare/v2.67.2..v2.67.1
 [2.67.1]: ../../compare/v2.67.1..v2.67.0

Dockerfile

@@ -50,9 +50,9 @@ FROM ocrd_core_base as ocrd_core_test
 ARG SKIP_ASSETS
 WORKDIR /build/core
 COPY Makefile .
+COPY .gitmodules .
 RUN if test -z "$SKIP_ASSETS" || test $SKIP_ASSETS -eq 0 ; then make assets ; fi
 COPY tests ./tests
-COPY .gitmodules .
 COPY requirements_test.txt .
 RUN pip install -r requirements_test.txt
 RUN mkdir /ocrd-data && chmod 777 /ocrd-data

Makefile

@@ -238,9 +238,9 @@ repo/assets repo/spec: always-update
 
 .PHONY: spec
 # Copy JSON Schema, OpenAPI from OCR-D/spec
-spec: repo/spec
-	cp repo/spec/ocrd_tool.schema.yml ocrd_validators/ocrd_validators/ocrd_tool.schema.yml
-	cp repo/spec/bagit-profile.yml ocrd_validators/ocrd_validators/bagit-profile.yml
+spec: # repo/spec
+	cp repo/spec/ocrd_tool.schema.yml src/ocrd_validators/ocrd_tool.schema.yml
+	cp repo/spec/bagit-profile.yml src/ocrd_validators/bagit-profile.yml
 
 #
 # Assets
@@ -273,7 +273,7 @@ test-logging: assets
 	cp src/ocrd_utils/ocrd_logging.conf $$tempdir; \
 	cd $$tempdir; \
 	$(PYTHON) -m pytest --continue-on-collection-errors -k TestLogging -k TestDecorators $(TESTDIR); \
-	rm -r $$tempdir/ocrd_logging.conf $$tempdir/.benchmarks; \
+	rm -r $$tempdir/ocrd_logging.conf $$tempdir/ocrd.log $$tempdir/.benchmarks; \
 	rm -rf $$tempdir/.coverage; \
 	rmdir $$tempdir
 
@@ -401,41 +401,3 @@ docker docker-cuda docker-cuda-tf1 docker-cuda-tf2 docker-cuda-torch:
 # Build wheels and source dist and twine upload them
 pypi: build
 	twine upload --verbose dist/ocrd-$(VERSION)*{tar.gz,whl}
-
-pypi-workaround: build-workaround
-	for dist in $(BUILD_ORDER);do twine upload dist/$$dist-$(VERSION)*{tar.gz,whl};done
-
-# Only in place until v3 so we don't break existing installations
-build-workaround: pyclean
-	cp pyproject.toml pyproject.toml.BAK
-	cp src/ocrd_utils/constants.py src/ocrd_utils/constants.py.BAK
-	cp src/ocrd/cli/__init__.py src/ocrd/cli/__init__.py.BAK
-	for dist in $(BUILD_ORDER);do \
-		cat pyproject.toml.BAK | sed "s,^name =.*,name = \"$$dist\"," > pyproject.toml; \
-		cat src/ocrd_utils/constants.py.BAK | sed "s,dist_version('ocrd'),dist_version('$$dist')," > src/ocrd_utils/constants.py; \
-		cat src/ocrd/cli/__init__.py.BAK | sed "s,package_name='ocrd',package_name='$$dist'," > src/ocrd/cli/__init__.py; \
-		$(MAKE) build; \
-	done
-	rm pyproject.toml.BAK
-	rm src/ocrd_utils/constants.py.BAK
-	rm src/ocrd/cli/__init__.py.BAK
-
-# test that the aliased packages work in isolation and combined
-test-workaround: build-workaround
-	$(MAKE) uninstall-workaround
-	for dist in $(BUILD_ORDER);do \
-		pip install dist/$$dist-*.whl ;\
-		ocrd --version ;\
-		make test ;\
-		pip uninstall --yes $$dist ;\
-	done
-	for dist in $(BUILD_ORDER);do \
-		pip install dist/$$dist-*.whl ;\
-	done
-	ocrd --version ;\
-	make test ;\
-	for dist in $(BUILD_ORDER);do pip uninstall --yes $$dist;done
-
-uninstall-workaround:
-	for dist in $(BUILD_ORDER);do $(PIP) uninstall --yes $$dist;done
-

README.md

@@ -47,17 +47,12 @@ complete stack of OCR-D-related software.
 
 The easiest way to install is via `pip`:
 
-```sh
-pip install ocrd
+    pip install ocrd
 
-# or just the functionality you need, e.g.
-
-pip install ocrd_modelfactory
-```
 
 All Python software released by [OCR-D](https://github.com/OCR-D) requires Python 3.8 or higher.
 
-**NOTE** Some OCR-D-Tools (or even test cases) _might_ reveal an unintended behavior if you have specific environment modifications, like:
+> **NOTE** Some OCR-D tools (or even test cases) _might_ reveal an unintended behavior if you have specific environment modifications, like:
 * using a custom build of [ImageMagick](https://github.com/ImageMagick/ImageMagick), whose format delegates are different from what OCR-D supposes
 * custom Python logging configurations in your personal account
 
@@ -82,7 +77,6 @@ Almost all behaviour of the OCR-D/core software is configured via CLI options an
 
 Some parts of the software are configured via environment variables:
 
-* `OCRD_METS_CACHING`: If set to `true`, access to the METS file is cached, speeding in-memory search and modification.
 * `OCRD_PROFILE`: This variable configures the built-in CPU and memory profiling. If empty, no profiling is done. Otherwise expected to contain any of the following tokens:
   * `CPU`: Enable CPU profiling of processor runs
   * `RSS`: Enable RSS memory profiling
@@ -95,18 +89,46 @@ Some parts of the software are configured via environment variables:
 * `XDG_CONFIG_HOME`: Directory to look for `./ocrd/resources.yml` (i.e. `ocrd resmgr` user database) – defaults to `$HOME/.config`.
 * `XDG_DATA_HOME`: Directory to look for `./ocrd-resources/*` (i.e. `ocrd resmgr` data location) – defaults to `$HOME/.local/share`.
 
-* `OCRD_DOWNLOAD_RETRIES`: Number of times to retry failed attempts for downloads of workspace files.
+* `OCRD_DOWNLOAD_RETRIES`: Number of times to retry failed attempts for downloads of resources or workspace files.
 * `OCRD_DOWNLOAD_TIMEOUT`: Timeout in seconds for connecting or reading (comma-separated) when downloading.
 
+* `OCRD_MISSING_INPUT`: How to deal with missing input files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed with next page's input
+  * `ABORT`: throw `MissingInputFile` exception
+
+* `OCRD_MISSING_OUTPUT`: How to deal with missing output files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed processing next page
+  * `COPY`: fall back to copying input PAGE to output fileGrp for page
+  * `ABORT`: re-throw whatever caused processing to fail
+
+* `OCRD_MAX_MISSING_OUTPUTS`: Maximal rate of skipped/fallback pages among all processed pages before aborting (decimal fraction, ignored if negative).
+
+* `OCRD_EXISTING_OUTPUT`: How to deal with already existing output files (for some fileGrp/pageId) during processing:
+  * `SKIP`: ignore and proceed processing next page
+  * `OVERWRITE`: force writing result to output fileGrp for page
+  * `ABORT`: re-throw `FileExistsError` exception
+
+
 * `OCRD_METS_CACHING`: Whether to enable in-memory storage of OcrdMets data structures for speedup during processing or workspace operations.
 
 * `OCRD_MAX_PROCESSOR_CACHE`: Maximum number of processor instances (for each set of parameters) to be kept in memory (including loaded models) for processing workers or processor servers.
 
+* `OCRD_MAX_PARALLEL_PAGES`: Maximum number of processor threads for page-parallel processing (within each Processor's selected page range, independent of the number of Processing Workers or Processor Servers). If set `>1`, then a METS Server must be used for METS synchronisation.
+
+* `OCRD_PROCESSING_PAGE_TIMEOUT`: Timeout in seconds for processing a single page. If set >0, when exceeded, the same as OCRD_MISSING_OUTPUT applies.
+
 * `OCRD_NETWORK_SERVER_ADDR_PROCESSING`: Default address of Processing Server to connect to (for `ocrd network client processing`).
 * `OCRD_NETWORK_SERVER_ADDR_WORKFLOW`: Default address of Workflow Server to connect to (for `ocrd network client workflow`).
 * `OCRD_NETWORK_SERVER_ADDR_WORKSPACE`: Default address of Workspace Server to connect to (for `ocrd network client workspace`).
 * `OCRD_NETWORK_RABBITMQ_CLIENT_CONNECT_ATTEMPTS`: Number of attempts for a worker to create its queue. Helpful if the rabbitmq-server needs time to be fully started.
 
+* `OCRD_NETWORK_CLIENT_POLLING_SLEEP`: How many seconds to sleep before trying `ocrd network client` again.
+* `OCRD_NETWORK_CLIENT_POLLING_TIMEOUT`: Timeout for a blocking `ocrd network client` (in seconds).
+
+* `OCRD_NETWORK_SOCKETS_ROOT_DIR`: The root directory where all mets server related socket files are created.
+* `OCRD_NETWORK_LOGS_ROOT_DIR`: The root directory where all ocrd_network related file logs are stored.
+
+
 
 ## Packages