Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: batch-map implementation #177

Merged
merged 17 commits into from
Jul 16, 2024
Merged

feat: batch-map implementation #177

merged 17 commits into from
Jul 16, 2024

Conversation

kohlisid
Copy link
Contributor

@kohlisid kohlisid commented Jul 3, 2024

Fixes #1788

kohlisid added 2 commits July 2, 2024 18:17
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Copy link

codecov bot commented Jul 3, 2024

Codecov Report

Attention: Patch coverage is 97.05882% with 6 lines in your changes missing coverage. Please review.

Project coverage is 94.82%. Comparing base (32a9d26) to head (f9d47a2).

Files Patch % Lines
pynumaflow/batchmapper/_dtypes.py 96.49% 4 Missing ⚠️
pynumaflow/batchmapper/async_server.py 94.44% 1 Missing ⚠️
pynumaflow/batchmapper/servicer/async_servicer.py 98.03% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #177      +/-   ##
==========================================
+ Coverage   94.65%   94.82%   +0.16%     
==========================================
  Files          52       56       +4     
  Lines        2003     2184     +181     
  Branches      119      130      +11     
==========================================
+ Hits         1896     2071     +175     
- Misses         78       83       +5     
- Partials       29       30       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
kohlisid added 5 commits July 10, 2024 16:32
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid changed the title feat: batch-map implementation feat: batch-map implementation [MERGE AFTER DOCKER ISSUE FIXED] Jul 11, 2024
@kohlisid kohlisid self-assigned this Jul 11, 2024
kohlisid added 2 commits July 11, 2024 13:49
Signed-off-by: Sidhant Kohli <[email protected]>
Signed-off-by: Sidhant Kohli <[email protected]>
"""
self.batch_mapper_instance: BatchMapCallable = batch_mapper_instance
self.sock_path = f"unix://{sock_path}"
self.max_threads = min(max_threads, int(os.getenv("MAX_THREADS", "4")))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's have a max cap on the threads. Too many is not good. Let's keep a cap at 16

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Defaulting to 4, putting a max cap at 16
Will change for all server types

Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from ab93 July 12, 2024 01:16
@ab93 ab93 marked this pull request as ready for review July 12, 2024 18:32
@ab93 ab93 requested review from vigith and whynowy as code owners July 12, 2024 18:32
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid changed the title feat: batch-map implementation [MERGE AFTER DOCKER ISSUE FIXED] feat: batch-map implementation Jul 15, 2024
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from ab93 July 15, 2024 03:28
@kohlisid kohlisid merged commit 921b7c0 into numaproj:main Jul 16, 2024
11 checks passed
@kohlisid kohlisid deleted the map-batch branch September 26, 2024 06:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Python UDF Server Implementation (numaflow-python)
2 participants