Implement BulkPublish for NATS JetStream
#3649
Draft
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit uses the BulkPublish API to introduce asynchronous publish. It uses NATS client `PublishAsync` to publish all messages, then iterates over all acks, gathers failed ones and returns. Signed-off-by: Tomasz Pietrek <[email protected]>
Signed-off-by: Tomasz Pietrek <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR implements
BulkPublishfor NATS JetStream with a goal to enable parallel processing of published messages.Continous async publishing would be a better solution, but there seem to be no interface for that in DAPR.
The alternatives:
PublishRequest.Metadataonpubsub.Publishto pass a switch that would enable asynchronous publishingNot sure what is the most idiomatic way to handle that in DAPR.
For some context: In future NATS will introduce actuall
BulkPublishwhich would be an atomic operation, so usingBulkPublishmight bring issues in the future, or at least rise the same question: how to switch between atomic and non atomic bulk publish.I would appreciate some guidance from the maintainers what is the best approach here, DAPR API wise.
Issue reference
#3115
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:
PR specific checklist