CORE-20867 Implement retry topic to handle persistent transient RPC Client errors #6385
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…-20867/retry-topic
…id replay logic and set request id correctly
…ct when not to save.
LWogan
requested review from
jasonbyrner3,
ronanbrowne,
rick-r3,
simon-johnson-r3,
blsemo,
Omar-awad,
aditisdesai,
vinir3,
vkolomeyko,
Sakpal,
owenstanford,
davidcurrie and
conalsmith-r3
as code owners
|
Jenkins build for PR 6385 build 14 Build Successful: |
...saging/src/main/kotlin/net/corda/messaging/api/mediator/config/EventMediatorConfigBuilder.kt
Outdated
Show resolved
Hide resolved
...ging/messaging-impl/src/main/kotlin/net/corda/messaging/mediator/processor/EventProcessor.kt
Outdated
Show resolved
Hide resolved
...ow-service/src/main/kotlin/net/corda/flow/messaging/mediator/FlowEventMediatorFactoryImpl.kt
Show resolved
Hide resolved
|
conalsmith-r3
approved these changes
Nov 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Design: https://github.com/corda/platform-eng-design/pull/658
API: corda/corda-api#1710
The current mediator messaging pattern in Corda can encounter an retry loop when transient errors are received from other Corda workers. This retry loop blocks flow topic partitions from progressing and it has been observed that the corda cluster affected can become permanently unstable due to the effects of consumer lag. This pattern is used by the flow worker to perform synchronous HTTP calls to various workers, including verification, token, crypto, uniqueness, and persistence workers.
To address this issue, a separate Kafka topic is dedicated to handling retries. This will allow the primary ingestion topics to continue processing unaffected flows, while introducing finite retry logic for flows impacted by transient errors.
Additionally AVRO version is bumped to fix a vulnerability