Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA]: Add new SimpleMessageBroker and supporting elements to NV-ingest #226

Closed
drobison00 opened this issue Nov 13, 2024 · 0 comments · Fixed by #238
Closed

[FEA]: Add new SimpleMessageBroker and supporting elements to NV-ingest #226

drobison00 opened this issue Nov 13, 2024 · 0 comments · Fixed by #238
Assignees
Labels
24.12 feature request New feature or request

Comments

@drobison00
Copy link
Collaborator

drobison00 commented Nov 13, 2024

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Currently preventing usage

Please provide a clear description of problem this feature solves

Overview:
Currently, the ingest pipeline relies on a message broker (Redis by default) to feed data, requiring the deployment of a message broker container and front-end REST service. For testing or proof of concept scenarios, it would be beneficial to have a more streamlined option that eliminates these dependencies.

Solution:
Implement a simple inline message broker that can be used within the pipeline and create a corresponding client interface. This will allow the ingest service to run independently, without requiring local dependencies on an external message broker or REST service.

Describe the feature, and optionally a solution or implementation and any alternatives

Introduce socket_task_source and socket_task_sink components to the nv_ingest service. These will be configurable to listen on a specified source, allowing jobs to be accepted and results returned over sockets. Additionally, update nv_ingest_client with options to submit jobs and fetch job results from the nv_ingest service via these socket connections.

Additional context

No response

@drobison00 drobison00 added the feature request New feature or request label Nov 13, 2024
@drobison00 drobison00 self-assigned this Nov 13, 2024
@sosahi sosahi added the 24.12 label Nov 14, 2024
@drobison00 drobison00 changed the title [FEA]: Add new 'socket source' to the pipeline to support testing without a message broker [FEA]: Add new SimpleMessageBroker and supporting elements to NV-ingest Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
24.12 feature request New feature or request
Projects
None yet
2 participants