-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extend the network client #1269
Merged
Merged
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit
Hold shift + click to select a range
2927a28
add integration test for client
MehmedGIT 8e7cd3e
fix the test dir path in docker
MehmedGIT bd16dd7
update network client
MehmedGIT b2c0675
integration test for client
MehmedGIT db6e566
Fix flag typo
MehmedGIT bec81ba
try docker host ip
MehmedGIT 4815896
remove the client server
MehmedGIT cb3460f
refactor status checks
MehmedGIT 920c1a9
fix test
MehmedGIT 2a843a8
fix: client processing request
MehmedGIT 3a238a7
add: client workflow run
MehmedGIT 50794f9
add timeout and wait to configs
MehmedGIT cc06fc3
Update src/ocrd_network/client_utils.py
MehmedGIT 4115937
refine status check methods
MehmedGIT 0136db0
add help for new env
MehmedGIT 734bbf0
add cli job status check
MehmedGIT f86bc23
add: help section to the cli
MehmedGIT 4194f9f
fix: required job id
MehmedGIT 97b3eea
add docstring to cli commands
MehmedGIT 8e7ba26
Fix: rename to block
MehmedGIT 69808b6
Fix: server_utils.py > 404 to 400
MehmedGIT 4de1e83
fix: set ps address if None in constructor
MehmedGIT d1af85b
fix: check report validation outside try block
MehmedGIT 50f73c5
fix: the annoying string dict
MehmedGIT 8f2861c
add: parameter_override
MehmedGIT 06a371c
add sort to network agents
MehmedGIT 4d85970
add: discovery cli, processors and processor
MehmedGIT bb3007d
add: check processing job log file
MehmedGIT ff4243f
fix: exception handling
MehmedGIT 5f746c1
ocrd network client: parse parameters and overrides
kba 8fc8bff
fix parameter parsing again
kba d73cfaa
Merge pull request #1270 from OCR-D/fix-parsing
MehmedGIT 15cea57
:memo: changelog
kba 18d743a
Merge branch 'master' into extend-network-client
kba 6608539
refactor client cli: process -> run
MehmedGIT File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When would this exception be raised? Validators should not raise errors but return a report. If you don't have a specific use case, it might be better to just not do
try/except
so that potential errors are actually raised and fixed.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wanted to be extra secure against unexpected errors. The workers or the server (network agents as a service) should ideally never crash due to some error. Especially with an HTTP 500 error on the client side and leaving the network agent in an unpredictable state. The exceptions from problematic requests are still logged.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Understood, but errors in
would mean that something is broken in our implementation or in jsonschema. So this would likely not be a fluke that happens once, we would need to stop processing and fix the bug at the source. The only variable factor is the
job_input.parameters
dict and if user-provided input breaks the validator - which it must absolutely never do - we should fix that in the validator.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A failing validator would also probably fail most of the processing/workflow jobs. Input from the user breaking down the entire infrastructure is conceptually wrong to me. Even if all the processing jobs fail, the Processing Server should still respond to other requests such as checking logs of old jobs. We would rather need a mechanism to prevent further submission of processing jobs in case X amount of jobs fail in a row - potentially preventing further requests only to the processing endpoint till it is fixed.
There is also currently no graceful shutdown for the processing server. I.e., once the server dies, anything inside the internal processing cache of the server (not the RabbitMQ) will be lost.