Code

Processing-Server#974

Code

Merged

kba merged 230 commits intomasterOCR-D/core:masterfrom dev-processing-brokerOCR-D/core:dev-processing-brokerCopy head branch name to clipboardApr 2, 2023

+2,799-81

Commits

Commits on Dec 7, 2022

• prototype for processing broker

  joschrew

  committed
  f2cf4e6

Commits on Dec 16, 2022

• refactoring: try to make it easier to read

  joschrew

  committed
  2a4919c

• adapt code to changed config-file

  joschrew

  committed
  d23bb3c

Commits on Dec 19, 2022

• change config representation class

  joschrew

  committed
  7e20fef

• add config validation

  joschrew

  committed
  d5a8641

• add deployment with docker-sdk

  joschrew

  committed
  fd8e724

Commits on Dec 20, 2022

• deploy queue and mongodb with docker-sdk

  joschrew

  committed
  8f0796d

• refactor code

  joschrew

  committed
  65dc60f

Commits on Jan 2, 2023

• Add the required dependencies

  MehmedGIT

  committed
  ad8acc6

• Make queue and db deploy more flexible

  MehmedGIT

  committed
  55ba0e4

• Refactor deployment of processors

  MehmedGIT

  committed
  22bc5de

• Conceptual integration of the RabbitMQ library

  MehmedGIT

  committed
  efdc5da

• Refactor: separate deplyoer from deployment utils

  MehmedGIT

  committed
  1724662

• Refactor processing broker/worker

  MehmedGIT

  committed
  16bcb4b

• Revert requirements.txt to keep track of future conflicting files

  MehmedGIT

  committed
  9901f90

• Merge remote-tracking branch 'origin/master' into dev-processing-broker

  MehmedGIT

  committed
  0ce527b

Commits on Jan 3, 2023

• Extend requirements

  MehmedGIT

  committed
  86979f2

• Adopt useful methods

  MehmedGIT

  committed
  07b1733

• Create network package by refactoring

  MehmedGIT

  committed
  6639c79

• Add example broker configuration

  MehmedGIT

  committed
  76c40a3

Commits on Jan 6, 2023

• Change logger names

  joschrew

  committed
  3e4e76c

• Replace double quoted strings with single quoted

  Show description for f1c27f3
  joschrew

  committed
  f1c27f3

Commits on Jan 9, 2023

• Add typehints for network files

  joschrew

  committed
  4fcaeb5

• Add address-option to processing broker

  joschrew

  committed
  d700522

• Remove informational non-code files again

  Show description for 019e94e
  joschrew

  committed
  019e94e

• Change functionsignature docker/ssh clientcreation

  joschrew

  committed
  36160cc

• Remove (not working) close_clients function

  joschrew

  committed
  23b0db9

Commits on Jan 10, 2023

• Move, rename and modify config-classes

  joschrew

  committed
  f218669

• Change enum value comparision for deploy_type

  joschrew

  committed
  078fa31

• Improve for loops over processors

  joschrew

  committed
  4f8d700

• Remove and change some asserts

  Show description for 6b549f1
  joschrew

  committed
  6b549f1

• Add a few explanatory comments

  joschrew

  committed
  f5ad740

• Change and and comments

  Show description for 02e3c4c
  joschrew

  committed
  02e3c4c

• Change config file validation and handling

  joschrew

  committed
  fde789a

Commits on Jan 11, 2023

• refactor: port -> ports_mapping

  MehmedGIT

  committed
  ab74eec

• refactor: _queue -> _rabbitmq

  MehmedGIT

  committed
  edaf55d

• Fix typo in kill_processing_workers

  MehmedGIT

  committed
  87a83a8

• Implement kill_processing_worker separately

  MehmedGIT

  committed
  e82de10

• Improve comments and checks

  MehmedGIT

  committed
  612dc0b

Commits on Jan 12, 2023

• Add processing worker cli template

  MehmedGIT

  committed
  76825fc

• Copy RabbitMQ utility library from the WebAPI impl repo

  MehmedGIT

  committed
  8c60172

• Add the reference note to the RabbitMQ utils

  MehmedGIT

  committed
  736a606

• Use actual RMQConsumer inside the processing_worker

  MehmedGIT

  committed
  1ec9371

• Use actual RMQPublisher inside the processing_broker

  MehmedGIT

  committed
  51690ee

• Refactor and optimize imports

  MehmedGIT

  committed
  af5c0b4

• Fix imports in rabbitmq_utils, add pika as a requirement

  MehmedGIT

  committed
  fb213e8

• Remove config file and tomli as a requirement

  MehmedGIT

  committed
  41defe3

• Refactor defaults in rabbitmq_utils webapi->network

  MehmedGIT

  committed
  03c0946

• Comment out RabbitMQ publisher/consumer

  MehmedGIT

  committed
  ca1b1b9

• Improve deploy/kill order

  MehmedGIT

  committed
  49089c7

• Split configurations for better readability

  MehmedGIT

  committed
  346bb2c

Commits on Jan 13, 2023

• Remove typing which was auto added by the IDE

  MehmedGIT

  committed
  1a290a2

• change scope of get_processor

  MehmedGIT

  committed
  1453b74

• Move methods from Processing Worker to Deployer

  MehmedGIT

  committed
  2933a06

• Refactor name -> processor_name

  MehmedGIT

  committed
  3300391

• Finish transfering of methods from Worker to Deployer

  MehmedGIT

  committed
  9a7641b

• Improve the template of processing worker

  MehmedGIT

  committed
  a059f1c

Commits on Jan 16, 2023

• Import the job and tool models from prev implementation

  MehmedGIT

  committed
  29c30e5

Commits on Jan 17, 2023

• Add processing worker wrappers for run_cli and run_processor

  MehmedGIT

  committed
  908b118

• Fix import of ocrd_messages

  MehmedGIT

  committed
  680b706

• RabbitMQ and MongoDB addr handling helpers

  MehmedGIT

  committed
  9de2175

• Add helpers for addr parsing

  MehmedGIT

  committed
  a2481bd

• Extend decorators to accept --queue and --database

  MehmedGIT

  committed
  c83fec9

• Extend processing worker

  MehmedGIT

  committed
  1d8c1b0

• Remove default values for --queue and --database

  MehmedGIT

  committed
  62ae01b

• Increase sleep command of the dummy docker container

  MehmedGIT

  committed
  eeb797b

• Fix rabbitmq path in logging of processing_worker

  MehmedGIT

  committed
  fd24c4f

• Refactor processing broker + activate queue creation

  MehmedGIT

  committed
  6359bdf

• Log messages received from the Management UI on port 15672

  MehmedGIT

  committed
  ef34e67

Commits on Jan 18, 2023

• Improve logging + add some TODOs and remarks

  MehmedGIT

  committed
  cfcc036

• Retrieve ocrd_tool by using util method

  MehmedGIT

  committed
  3bd6f70

• Force logging settings - paramiko=INFO, ocrd.network=DEBUG

  MehmedGIT

  committed
  7138979

• Implement on_consumed_message method

  MehmedGIT

  committed
  0f826fd

• default message constructing methods

  MehmedGIT

  committed
  2a5d442

• Append default global vhost, when not provided

  MehmedGIT

  committed
  7912ae3

• database prefix should be passed as an argument

  MehmedGIT

  committed
  fb11953

• Fix --queue and --database, not flags

  MehmedGIT

  committed
  e93d702

• Add remarks about start_*_processor methods

  MehmedGIT

  committed
  952e6e9

Commits on Jan 19, 2023

• Provide a basic testing endpoint to processing broker (aka server)

  MehmedGIT

  committed
  43050ab

Commits on Jan 20, 2023

• Make address param for processing broker required

  joschrew

  committed
  ae18c08

• Use credentials from config for rabbitmq messaging

  Show description for fdfaf8d
  joschrew

  committed
  fdfaf8d

• Add new parameters to processor help output

  joschrew

  committed
  f387f92

• Improve error message for processing worker

  joschrew

  committed
  d35d006

Commits on Jan 23, 2023

• Insert credentials on rabbitmq startup

  Show description for beca781
  joschrew

  committed
  beca781

• Change message exchange from pickle to str for now

  Show description for f2c3e6e
  joschrew

  committed
  f2c3e6e

Commits on Jan 24, 2023

• Change rabbitmq default credentials

  Show description for 2f045b3
  joschrew

  committed
  2f045b3

• Add optional path_to_bin_dir to config and use it

  Show description for d1c1f50
  joschrew

  committed
  d1c1f50

Commits on Jan 25, 2023

• Add route to p-broker to start a p-worker

  joschrew

  committed
  344e593

• Add typehints for a method in processing_worker

  joschrew

  committed
  77a316d

• Replace double quoted strings with single quoted

  joschrew

  committed
  e16e3d4

• Add missing typehints for network modules

  joschrew

  committed
  512ea77

• Review (mainly comments for) CustomDockerClient

  Show description for e876cf9
  joschrew

  committed
  e876cf9

Commits on Jan 26, 2023

• Remind myself to rename the broker

  joschrew

  committed
  59e8418

• Reuse parts of older impl for running a processor

  Show description for 5900c73
  joschrew

  committed
  5900c73

• Add processor enpoints from previous impl (pr 884)

  joschrew

  committed
  2323fac

• Change comments and format and do some cleanup

  joschrew

  committed
  c57e75c

Commits on Jan 30, 2023

• Adapt to latest config changes

  joschrew

  committed
  868d58d

• Use definitions.json for rabbitmq

  joschrew

  committed
  53e9b47

• Rename processing broker to processing server

  joschrew

  committed
  72a8e5c

• Set job state after worker finishes

  joschrew

  committed
  0f399ea

• Add enpoint to list processors

  joschrew

  committed
  8143614

Commits on Feb 2, 2023

• Remove redundant log output

  joschrew

  committed
  871323b

• Ensure processor availability before processing

  joschrew

  committed
  7f4f1d7

• Implement the result-queue logic - worker uses RMQPublisher

  MehmedGIT

  committed
  1caffd6

• Send encoded result message

  MehmedGIT

  committed
  a16aa02

Commits on Feb 8, 2023

• Add exception handler to debug validation

  MehmedGIT

  committed
  ab326f7

Commits on Feb 9, 2023

• Workers push finished job status to result queue

  MehmedGIT

  committed
  4368b53

• Example NF script (still broken) executed by the WF Server to submit jobs to Processing Server

  MehmedGIT

  committed
  e2452e4

Commits on Feb 10, 2023

• Add workspace_id parameter to processor-call

  joschrew

  committed
  de3b41a

• Add workspace model basically that from webapi

  joschrew

  committed
  8b583d9

• Add workspace_id/path param-check to run_processor

  joschrew

  committed
  5d218a1

• Resolve mets path with workspace id

  joschrew

  committed
  1142449

• Check off some trivial todos

  joschrew

  committed
  be0fe51

• Stop splitting config in ProcessingServer

  Show description for 67753b1
  joschrew

  committed
  67753b1

• Stop storing pid for queue and mongo in config

  joschrew

  committed
  6a526af

• Separating config and data for hosts

  Show description for d9d959e
  joschrew

  committed
  d9d959e

Commits on Feb 13, 2023

• Removing unnecessary things after meeting with Jonas

  MehmedGIT

  committed
  f51b3b8

Commits on Feb 14, 2023

• Add cleanup and exceptionhandling to startup

  Show description for 30498a7
  joschrew

  committed
  30498a7

• Remove unnecessary loggging and clean comments

  joschrew

  committed
  6a3836d

• Add job response model

  Show description for c838ed7
  joschrew

  committed
  c838ed7

• Add mechanism to ensure queue startup

  joschrew

  committed
  d763d4c

• Remove a few more unneeded todo-notes

  joschrew

  committed
  5b4e22d

Commits on Feb 16, 2023

• Merge branch 'master' into dev-processing-broker

  Show description for f6fe1c3
  MehmedGIT

  committed
  f6fe1c3

• Remove unnecessary parts

  MehmedGIT

  committed
  84acd77

• Add logging to file for processing workers

  MehmedGIT

  committed
  b586f70

Commits on Feb 17, 2023

• Wait 100ms between deployment of ocr-d processors

  MehmedGIT

  committed
  0bb9bdb

• Merge master

  MehmedGIT

  committed
  ec9ffc6

• Dirty fix - disable interactive shell logging for calamari

  MehmedGIT

  committed
  9ab1146

• Enable instance caching for processors - tested and working

  MehmedGIT

  committed
  6cd2a89

• Add tensorflow to core requirements

  Show description for 144f262
  MehmedGIT

  authored
  144f262

Commits on Feb 20, 2023

• Add changes from fix-972 branch

  MehmedGIT

  committed
  6c3c6d5

• Replace ocrd logging with python logging

  MehmedGIT

  committed
  cb54a67

• Lower the scope of tf keras import

  MehmedGIT

  committed
  5a328c2

• Revert logging back to ocrd logging in worker

  MehmedGIT

  committed
  c02f0ba

Commits on Feb 21, 2023

• Refactor imports, logging, spacing

  MehmedGIT

  committed
  cef1d0d

• Remove config-var path_to_bin_dir again

  joschrew

  committed
  b73c899

• Rearrange docker and ssh client creation

  joschrew

  committed
  3135a4b

Commits on Feb 22, 2023

• Fix copy paste mistake from last commit

  joschrew

  committed
  c00dc04

• Add workaround for rabbitmq startup on remote host

  joschrew

  committed
  bd3a6c2

Commits on Feb 23, 2023

• Remove the unnecessary rabbitmq.conf file

  MehmedGIT

  committed
  a1d5611

Commits on Feb 24, 2023

• Rename table for workspace in mongo

  joschrew

  committed
  de492b8

• Use config credentials for server and workers

  MehmedGIT

  committed
  6c09711

• Create ocrd_network package

  joschrew

  committed
  b264652

• Remove usage of definitions.json with rabbitmq

  joschrew

  committed
  826d9a0

• Fix dependency in requirements

  MehmedGIT

  committed
  ad98e12

Commits on Feb 27, 2023

• Update doc for processing server

  joschrew

  committed
  fffb7dc

Commits on Mar 2, 2023

• Improve exception message

  Show description for f5e45b3
  

  MehmedGIT

  and

  bertsky

  authored
  f5e45b3

• Remove parsing of stdout in ocrd decorators

  Show description for 9829b18
  

  MehmedGIT

  and

  bertsky

  authored
  9829b18

• Update processing worker cli help

  Show description for b73e99a
  

  MehmedGIT

  and

  bertsky

  authored
  b73e99a

• Update ocrd-network package info in __init__

  MehmedGIT

  authored
  cf7321d

• Update ocrd_network/ocrd_network/deployer.py info

  Show description for 132b93f
  

  MehmedGIT

  and

  bertsky

  authored
  132b93f

• Change getcwd() call location

  MehmedGIT

  authored
  a2a7fe9

• Update ocrd_network/ocrd_network/deployer.py help

  Show description for 22311f1
  

  MehmedGIT

  and

  bertsky

  authored
  22311f1

• Update ocrd_network/ocrd_network/deployer.py

  Show description for f9c5414
  

  MehmedGIT

  and

  bertsky

  authored
  f9c5414

• Raise the error instead of returning it

  MehmedGIT

  committed
  4e66ed3

• Revert getcwd() location - failing tests

  MehmedGIT

  committed
  d1f287d

• Fix set_job_state

  MehmedGIT

  committed
  ddc92d7

Commits on Mar 3, 2023

• Make conditional TF import

  MehmedGIT

  committed
  75b5d4e

Commits on Mar 9, 2023

• Remove unused function

  Show description for b3ada1f
  joschrew

  committed
  b3ada1f

Commits on Mar 10, 2023

• Implement custom type check

  MehmedGIT

  committed
  f154949

Commits on Mar 15, 2023

• Change native processor startup output handling

  Show description for 8a256d5
  joschrew

  committed
  8a256d5

• Fix uvicorn logging bug

  Show description for e125a14
  joschrew

  committed
  e125a14

• Add possibility to run bashlib processors too

  joschrew

  committed
  b5d3496

Commits on Mar 16, 2023

• Add module docstring for database

  joschrew

  committed
  5ff8860

• parameters -> arguments in RMQ connector

  MehmedGIT

  committed
  2b5cc3b

• Fix quotes

  MehmedGIT

  committed
  ad04cb6

• better error message

  MehmedGIT

  committed
  8e7b1f8

Commits on Mar 17, 2023

• Improve validation and parsing of URIs

  MehmedGIT

  committed
  44e1eaa

• Clean, refactor, and add small TODOs

  MehmedGIT

  committed
  c791260

Commits on Mar 20, 2023

• chdir to ws as in #987

  MehmedGIT

  committed
  494c0ab

• Call getcwd before get_processor

  MehmedGIT

  committed
  d784a75

• Set ws outside the constructor

  MehmedGIT

  committed
  43a5660

Commits on Mar 21, 2023

• Modify OcrdResultMessage

  MehmedGIT

  committed
  e396b2b

• Move check before assignments

  MehmedGIT

  committed
  57d4791

• Remove pymongo, add call_sync wrapper

  MehmedGIT

  committed
  7288d6d

• Refactor helpers -> utils

  MehmedGIT

  committed
  5e6d0ad

• Add callback_url, improve models

  MehmedGIT

  committed
  66851d4

• Remove the debugging file

  MehmedGIT

  committed
  07828a6

• Remove result queue related info logs

  MehmedGIT

  committed
  0eb74b9

Commits on Mar 22, 2023

• Initiate DB client after db addr verification

  MehmedGIT

  committed
  da2e7e1

• Improve exception message

  MehmedGIT

  committed
  31d3a08

• refactor result_queue, callback_url checks

  MehmedGIT

  committed
  b82446c

• Set workspace_id when available

  MehmedGIT

  committed
  0f0a6d3

• minor variable and comment improvements

  MehmedGIT

  committed
  44b6b6e

• fix result queue related things

  MehmedGIT

  committed
  8e0d7d8

• path -> path_to_mets in Job

  MehmedGIT

  committed
  d1e7ff9

• Fix job.path -> job.path_to_mets, refactor

  MehmedGIT

  committed
  c6be974

• Create job_id, don't rely on document_id

  MehmedGIT

  committed
  4dd0ea9

• Refactor status -> state to have same standard

  MehmedGIT

  committed
  69399fb

• Remove json.dumps

  MehmedGIT

  committed
  d1d85c6

• Refactor the setup.py

  MehmedGIT

  committed
  93b7f5b

• Big refactoring - use relative imports

  MehmedGIT

  committed
  8f32004

• Fix setup.py

  MehmedGIT

  committed
  3c3273b

• Merge master v2.48.0

  MehmedGIT

  committed
  32b935b

Commits on Mar 23, 2023

• Refactor processing config validation

  MehmedGIT

  committed
  e080a0f

• Validate processing message against message schema

  MehmedGIT

  committed
  59a06ab

Commits on Mar 24, 2023

• Fix: catch db workspace exception

  MehmedGIT

  committed
  787510e

• impove: match the returned http status code

  MehmedGIT

  committed
  1679382

• fix: resolve mets path in worker, not server

  MehmedGIT

  committed
  709c46c

• fix: process job model

  MehmedGIT

  committed
  39840e2

• db single update method

  MehmedGIT

  committed
  9ba05e0

• calculate execution time in ms

  MehmedGIT

  committed
  0fe51f2

• resolve: suggestions by kba

  MehmedGIT

  committed
  9910482

• bashlib: implement Processing Worker args

  bertsky

  authored
  4774f51

• improve generate_processor_help:

  Show description for 6741d55
  bertsky

  authored
  6741d55

• ocrd_cli_options: remove redundant help kwarg

  Show description for 0ccc963
  bertsky

  authored
  0ccc963

• start native cli: no need for is_bashlib_processor

  bertsky

  authored
  e2e6e69

• rm is_bashlib_processor

  bertsky

  authored
  4ab2657

• rm Py36 test – no longer supported

  bertsky

  authored
  a4b9c96

• CI: use Python images from CircleCI instead of DH

  Show description for 3f13747
  bertsky

  authored
  3f13747

• require Py37+

  bertsky

  authored
  6ead599

• require Py37+

  bertsky

  authored
  89570a7

• require Py37+

  bertsky

  authored
  e1f6ed8

• require Py37+

  bertsky

  authored
  fed4128

• require Py37+

  bertsky

  authored
  af08d1e

• require Py37+

  bertsky

  authored
  21c46c1

Commits on Mar 25, 2023

• fix CI:

  Show description for fe3303e
  bertsky

  authored
  fe3303e

Commits on Mar 26, 2023

• --help: clearer description of --page-id

  Show description for df7bfee
  

  kba

  and

  bertsky

  authored
  df7bfee

• .circleci/config.yml: fix syntax

  kba

  committed
  378400e

• .circleci/config.yml: fix syntax

  kba

  committed
  dbae7c4

• Merge branch 'dev-processing-broker-add-bashlib-processing-worker' of https://github.com/OCR-D/core into dev-processing-broker-add-bashlib-processing-worker

  kba

  committed
  53f94b7

• .circleci/config.yml: fix syntax

  kba

  committed
  25cc6fc

• .circleci/config.yml: fix syntax

  kba

  committed
  176cf73

• Merge pull request #1024 from OCR-D/dev-processing-broker-add-bashlib-processing-worker

  Show description for eeaafc4
  MehmedGIT

  authored
  eeaafc4

Commits on Mar 27, 2023

• Make receiving job info for procesor work again

  joschrew

  committed
  2fa674f

• provide flexible queue checks

  MehmedGIT

  committed
  2fc3856

• remove unnecessary checks

  MehmedGIT

  committed
  7835154

• bruh, I need some rest for today

  Show description for 123366b
  MehmedGIT

  authored
  123366b

Commits on Mar 28, 2023

• check defaults, pass shallow copy

  MehmedGIT

  committed
  6fac7b8

Commits on Apr 2, 2023

• Update README.md

  Show description for bc9a29a
  

  kba

  and

  bertsky

  authored
  bc9a29a