Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨ Aiohttp long running tasks client #3290

Conversation

sanderegg
Copy link
Member

What do these changes do?

Related issue/s

related to ITISFoundation/osparc-issues#558

How to test

cd packages/service-library
make "install-dev[all]"
make "test-dev[all]"

Checklist

@sanderegg sanderegg self-assigned this Aug 29, 2022
@sanderegg sanderegg requested a review from pcrespov as a code owner August 29, 2022 14:08
@sanderegg sanderegg added this to the Brutalism+1 milestone Aug 29, 2022
@sanderegg sanderegg force-pushed the enhancement/aiohttp_long_running_tasks_client branch from f99512d to dcb708f Compare August 29, 2022 14:10
@codecov
Copy link

codecov bot commented Aug 29, 2022

Codecov Report

Merging #3290 (353c035) into master (6f7a4e1) will decrease coverage by 0.9%.
The diff coverage is 89.9%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #3290     +/-   ##
========================================
- Coverage    82.2%   81.3%   -1.0%     
========================================
  Files         761     762      +1     
  Lines       32544   32629     +85     
  Branches     4160    4167      +7     
========================================
- Hits        26781   26551    -230     
- Misses       4903    5230    +327     
+ Partials      860     848     -12     
Flag Coverage Δ
integrationtests 63.4% <100.0%> (-2.8%) ⬇️
unittests 78.8% <89.9%> (+<0.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...library/src/servicelib/long_running_tasks/_task.py 95.2% <ø> (ø)
...c/servicelib/aiohttp/long_running_tasks/_server.py 89.5% <77.2%> (-10.5%) ⬇️
...rc/servicelib/aiohttp/long_running_tasks/client.py 92.8% <92.8%> (ø)
...rc/servicelib/aiohttp/long_running_tasks/server.py 100.0% <100.0%> (ø)
...rc/simcore_service_webserver/long_running_tasks.py 100.0% <100.0%> (ø)
...rvice_webserver/projects/projects_handlers_crud.py 88.8% <100.0%> (+1.5%) ⬆️
...simcore_service_webserver/computation_subscribe.py 27.0% <0.0%> (-61.0%) ⬇️
...k/src/simcore_sdk/node_ports_common/filemanager.py 27.5% <0.0%> (-55.8%) ⬇️
...src/simcore_sdk/node_ports_common/file_io_utils.py 33.6% <0.0%> (-54.6%) ⬇️
.../simcore_service_webserver/computation_settings.py 55.5% <0.0%> (-44.5%) ⬇️
... and 20 more

Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Please consider my request below.

@sanderegg sanderegg force-pushed the enhancement/aiohttp_long_running_tasks_client branch from 13afd5b to 164e57d Compare August 30, 2022 09:55
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very useful feature!
I have some comments/thoughts regarding the design and a couple of questions

NOTE: I guess as soon as this client interface gets stable it will be also extended to fastapi-based servers, right?

:yield: a task containing the current TaskProgress
:rtype: Iterator[AsyncGenerator[LRTask, None]]
"""
task = None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, this function encapsulates logic with four possible requests, namely

  • _start
  • _waif_for_completion
  • _task_result
  • _abort_task

Each of the above can potentially raise ClientResponseError, therefore the exceptions interface here remains ClientResonseError. Q1: how does the caller know which of these requests above errored? Would it make sense to define an error wrapper that associates ClientResonseError upon start, while completion, etc??

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, I think the only where it might make sense to differentiate is the start. the rest shall behave as usual. as this is wrapped, I do not think the caller might know what is going on with the long running task internals.

session: ClientSession,
url: URL,
json: Optional[RequestBody] = None,
wait_timeout_s: int = 1 * _HOUR,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These timing options are related with the polling interval. I would rather reuse tenacity kwargs. This way the caller has far more flexibility in the way it pools (e.g. it might poll the state with an exponential/random delay etc)

I have some suggestions regarding the name as well below (check the test)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, the wait_timeout is related with how long the client wants to wait, a.k.a ClientTimeout. will rename it.

The interval one, I want to change that later by using a Retry-After header. potentially different from the one of tenacity

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, I changed these. please have a look.
I kept the client_timeout which makes sense I think

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@sanderegg sanderegg merged commit c604d7c into ITISFoundation:master Aug 30, 2022
@sanderegg sanderegg deleted the enhancement/aiohttp_long_running_tasks_client branch August 30, 2022 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants