Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Standalone Processor Server (#884 + #974) #1029

Closed
wants to merge 232 commits into from
Closed

Standalone Processor Server (#884 + #974) #1029

wants to merge 232 commits into from

Conversation

MehmedGIT
Copy link
Contributor

This draft PR is supposed to get important aspects from #884 and adapt to #974 so it can extend the ocrd_network package to provide the standalone Processor Server feature.

I am creating this a bit early so it's easy to follow what is going on with this. Still broken and fixes are needed. A lot more refactoring is needed to follow afterward.

joschrew and others added 30 commits December 7, 2022 16:25
@joschrew
prototype for processing broker
f2cf4e6
@joschrew
refactoring: try to make it easier to read
2a4919c
@joschrew
adapt code to changed config-file
d23bb3c
@joschrew
change config representation class
7e20fef
@joschrew
add config validation
d5a8641
@joschrew
add deployment with docker-sdk
fd8e724
@joschrew
deploy queue and mongodb with docker-sdk
8f0796d
@joschrew
refactor code
65dc60f
@MehmedGIT
Add the required dependencies
ad8acc6
@MehmedGIT
Make queue and db deploy more flexible
55ba0e4
@MehmedGIT
Refactor deployment of processors
22bc5de
@MehmedGIT
Conceptual integration of the RabbitMQ library
efdc5da
@MehmedGIT
Refactor: separate deplyoer from deployment utils
1724662
@MehmedGIT
Refactor processing broker/worker
16bcb4b
@MehmedGIT
Revert requirements.txt to keep track of future conflicting files
9901f90
@MehmedGIT
Merge remote-tracking branch 'origin/master' into dev-processing-broker
0ce527b
@MehmedGIT
Extend requirements
86979f2
@MehmedGIT
Adopt useful methods
07b1733
@MehmedGIT
Create network package by refactoring
6639c79
@MehmedGIT
Add example broker configuration
76c40a3
@joschrew
Change logger names
3e4e76c
@joschrew
Replace double quoted strings with single quoted
f1c27f3 
We decided to use only single quotes for strings to make it consistent.
I kept docstrings in triple double quotes because of PEP 257.
@joschrew
Add typehints for network files
4fcaeb5
@joschrew
Add address-option to processing broker
d700522
@joschrew
Remove informational non-code files again
019e94e 
They were added to give additional information but are not needed here
any more in this place
@joschrew
Change functionsignature docker/ssh clientcreation
36160cc
@joschrew
Remove (not working) close_clients function
23b0db9
@joschrew
Move, rename and modify config-classes
f218669
@joschrew
Change enum value comparision for deploy_type
078fa31
@joschrew
Improve for loops over processors
4f8d700
bertsky and others added 19 commits March 25, 2023 00:34
@bertsky
require Py37+
fed4128
@bertsky
require Py37+
af08d1e
@bertsky
require Py37+
21c46c1
@bertsky
fix CI:
fe3303e 
- cimg needs sudo
- don't specify Python subminors, so it always picks latest
- activate docker_layer_caching for deploy to speed up
@kba @bertsky
--help: clearer description of --page-id
df7bfee 
Co-authored-by: Robert Sachunsky <[email protected]>
@kba
.circleci/config.yml: fix syntax
378400e
@kba
.circleci/config.yml: fix syntax
dbae7c4
@kba
Merge branch 'dev-processing-broker-add-bashlib-processing-worker' of h…
53f94b7 
…ttps://github.com/OCR-D/core into dev-processing-broker-add-bashlib-processing-worker
@kba
.circleci/config.yml: fix syntax
25cc6fc
@kba
.circleci/config.yml: fix syntax
176cf73
@MehmedGIT
Merge pull request #1024 from OCR-D/dev-processing-broker-add-bashlib…
eeaafc4 
…-processing-worker

add bashlib processing worker, require Python 3.7
@joschrew
Make receiving job info for procesor work again
2fa674f
@MehmedGIT
provide flexible queue checks
2fc3856
@MehmedGIT
remove unnecessary checks
7835154
@MehmedGIT
basics of the procesor server
1c37ad5
@MehmedGIT
Adapt the fix from #974
42f16ab
@MehmedGIT
check defaults, pass shallow copy
ba92c87
@MehmedGIT
complete processor server internals
0644b9e
@MehmedGIT
implement most of the logic for everything
cdb73d5
@MehmedGIT MehmedGIT requested review from kba, tdoan2010, bertsky and joschrew and removed request for kba, tdoan2010, bertsky and joschrew March 28, 2023 23:27
@bertsky
Copy link
Collaborator

bertsky commented Mar 28, 2023

Excellent! For a proper review, it would help to see the actual diff over dev-processing-broker. I think it would make more sense to start a new PR against that branch (even if it's going to be merged independently). After reviewing and working on the branch, you could still open a PR against master.

@MehmedGIT
Copy link
Contributor Author

MehmedGIT commented Mar 29, 2023 •
edited
Loading

Excellent! For a proper review, it would help to see the actual diff over dev-processing-broker. I think it would make more sense to start a new PR against that branch (even if it's going to be merged independently). After reviewing and working on the branch, you could still open a PR against master.

I just wanted to assign reviewers but it also automatically requests a review. This is still far away from being reviewed... You're right. I am going to create a PR to #974. Check #1030.

@MehmedGIT MehmedGIT closed this Mar 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@MehmedGIT @bertsky @joschrew @kba