Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: class based refactor for SDK #129

Merged
merged 78 commits into from
Jan 24, 2024
Merged

feat: class based refactor for SDK #129

merged 78 commits into from
Jan 24, 2024

Conversation

kohlisid
Copy link
Contributor

@kohlisid kohlisid commented Jan 9, 2024 •
edited
Loading

Fixes #106

  1. Refactor the SDKs to accept both classes and function as handlers
  2. Seperated common server utilities
  3. Abstracted server start call to have common function for all server types, do not need to use aiorun seperately for async functions. Makes it easier to switch between them for the UDF owner.
  4. Different server classes for each UDF
  5. Enable UV loop for async execution

kohlisid added 30 commits January 4, 2024 11:42
@kohlisid
refactor init
e4df766 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
init
382bdf2 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
move infor
87420ca 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
move infor
136da24 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
await change
58b6496 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
await change
83d366a 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
change to create task
ceee882 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
change to create task
b828204 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
ev loop
837de07 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
cleanup
c4f0376 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
cleanup
b385e25 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
cleanup
a22673b 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
server on same thread
fe52e9f 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
server on same thread
e2c315d 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
server on same thread
28b5c70 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
Export class
da9d1db 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
Class refactor
27967bf 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
multiproc info fix
a0ac382 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
modular
05a2cfe 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
modular
3f6049f 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
modular
b642b17 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
modular
40bb295 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
mapstream
e3b7090 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
mapstream
88bb474 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
mapstream
e187f90 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
reduce
d7ff4a1 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
reduce
32d2975 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
sink
3e3238a 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
sink
e35b156 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
sink
1ba673d 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
examples
6bf2c4a 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
README
6e44a6c 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
REDUCER INSTANCE
cbf47bf 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
REDUCER INSTANCE
d88ed5c 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid marked this pull request as draft January 18, 2024 02:53
@kohlisid
deep copy
3961f95 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
deep copy
4065717 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
deep copy
d2c844d 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid
lint
d3b59ff 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from vigith January 22, 2024 20:01
@kohlisid
add reducer test
6b56383 
Signed-off-by: Sidhant Kohli <[email protected]>
vigith
vigith reviewed Jan 22, 2024
View reviewed changes
examples/map/multiproc_map/example.py
Comment on lines +40 to +42
prime_class = PrimeMap()
# Server count is the number of server processes to start
grpc_server = MapMultiprocServer(prime_class, server_count=server_count)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
prime_class = PrimeMap()
# Server count is the number of server processes to start
grpc_server = MapMultiprocServer(prime_class, server_count=server_count)
# Server count is the number of server processes to start
# non-thread safe
grpc_server = MapMultiprocServer(instance=PrimeMap(), server_count=server_count)
# thread safe since we create an instance for every message
grpc_server = MapMultiprocServer(class=PrimeMap, server_count=server_count)

don't implement but just think

@kohlisid
change reduce signature
02afc01 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from vigith January 23, 2024 06:43
@kohlisid kohlisid marked this pull request as ready for review January 23, 2024 17:49
ab93
ab93 reviewed Jan 23, 2024
View reviewed changes
Copy link
Member

@ab93 ab93 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments!

examples/map/forward_message/example.py Outdated Show resolved Hide resolved
examples/map/forward_message/example.py Outdated Show resolved Hide resolved
examples/reduce/counter/example.py Outdated Show resolved Hide resolved
examples/sideinput/simple-sideinput/example.py Outdated Show resolved Hide resolved
examples/sink/async_log/example.py Outdated Show resolved Hide resolved
pynumaflow/mapper/_dtypes.py Outdated Show resolved Hide resolved
pynumaflow/mapper/async_server.py Show resolved Hide resolved
pynumaflow/mapper/multiproc_server.py Show resolved Hide resolved
pynumaflow/reducer/_dtypes.py Outdated Show resolved Hide resolved
@kohlisid
comments
d8bcefe 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from ab93 January 23, 2024 20:03
ab93
ab93 requested changes Jan 23, 2024
View reviewed changes
examples/sink/async_log/example.py Outdated Show resolved Hide resolved
@kohlisid
comments
d8ce1ce 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid requested a review from ab93 January 24, 2024 02:08
vigith
vigith approved these changes Jan 24, 2024
View reviewed changes
examples/reduce/counter/Makefile Outdated Show resolved Hide resolved
pynumaflow/reducer/async_server.py Outdated Show resolved Hide resolved
pynumaflow/reducer/async_server.py Outdated Show resolved Hide resolved
@kohlisid kohlisid mentioned this pull request Jan 24, 2024
Merged
@kohlisid
comments
59d6550 
Signed-off-by: Sidhant Kohli <[email protected]>
@kohlisid kohlisid merged commit 39ec103 into numaproj:main Jan 24, 2024
10 checks passed
@kohlisid kohlisid deleted the class-refactor branch February 28, 2024 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vigith vigith vigith approved these changes

@ab93 ab93 ab93 approved these changes

@whynowy whynowy Awaiting requested review from whynowy whynowy is a code owner

Assignees

@kohlisid kohlisid

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Class based approach for passing the handler
3 participants
@kohlisid @vigith @ab93