CORE-19136: Merging forward updates from release/os/5.1 to release/interop/syntax - 2024-02-07 #5639
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![corda-jenkins-ci02[bot]](https://avatars.githubusercontent.com/u/22600631?s=60&v=4){kind=small}
* Temporary revert Kafka config changes to test impact on CPU load. * Set fetch.max.wait.ms = 500 * Revert kafka configs back to release version, except from `fetch.max.wait.ms` * Increase mediator pull frequency.
Update event mediator so it uses the State returned by the processor, instead of the original one, when making the decision on whether to delete it from the persistent storage or not. Failing to do so leaves orphaned States in the persistent storage, which in turn ends up causing other issues in the system. --------- Co-authored-by: James Higgs <[email protected]>
…flakiness (#5064) Comparing the timestamp between requests and responses was, in some cases, causing test failures if the times were too close together. Moving to a more granular comparison to prevent this.
…iator (#5069) Ensures that states are stored before publishing events back to Kafka. By doing these operations in this order, the messaging layer ensures that any visible effects outside the processor being invoked do not become apparent until after a record of the current state has been made. This ensures that certain types of race conditions or ledger mutation bugs cannot occur. This change also fixes missing synchronization on the data structure holding output events, which is required when processing in a multi-threaded environment. This can manifest as lost messages when placed under load.
…FatalException (#5044) Change: This PR will implement checkpoint cleanup handler when processing FlowFatalException. Reasoning: In order to address CORE-17966: When exceptions such as FlowFatalException are thrown in the Flow Engine the exception handlers cleanup the checkpoints and schedule cleanup of the session states. However they currently do not cleanup the StartFlow Flow Mapper state for RPC started flows the checkpoint cleanup handler will be integrated in a series of small PRs.
…5076) * CORE-18088 Synchronized registration/removal of endpoints in JavalinServer * CORE-18088 Using mutable set for "endpoints" in JavalinServer since access is now synchronized with "serverLock"
This change adds a prefix to consumer group IDs to support multi-tenancy (>1 Corda cluster within a single Kafka cluster), in the same way that we currently do for topic names. Without this, consumers in different Corda deployments were being considered to be part of the same consumer group; this causes unnecessary re-balances, and was logging authz exceptions on partition assignment.
#5078) The issue was that on forced config reconciliation we would reconcile all config sections and therefore propagate them across the cluster no matter if their config value actually changed or not as part of the defaulting process. This caused components across the cluster to restart on db-worker restarts (one of which is the http server within the rest-worker which was causing its unavailability). * Contains the above config propagation on the db-worker side. On forced reconciliation checks if after applying the defaulting process to a config value read from the DB, it changes (expands) compared to its Kafka corresponding one and only then reconciles it
Sort the returned states before building the filters to use so that the lower bound is always older in time than the upper bound.
…MarkedForKillException (#5057) Change: This PR will implement checkpoint cleanup handler when processing FlowMarkedForKillException. Reasoning: In order to address CORE-17966: When exceptions such as FlowFatalException are thrown in the Flow Engine the exception handlers cleanup the checkpoints and schedule cleanup of the session states. However they currently do not cleanup the StartFlow Flow Mapper state for RPC started flows the checkpoint cleanup handler will be integrated in a series of small PRs.
Add token-selection section to local deployment
…and (#5114) * CORE-18357: Allow old version of SetOwnRegistrationStatus in the command * Set to the correct version
The query for `findUnconsumedStatesByExactType` did not actually filter by `consumed IS NULL` so it returned all states regardless of if they were consumed or not. Added `consumed IS NULL` to fix this.
Makes `MemberInfo` serializable by registering custom serializers for `MemberContext` and `MGMContext`. The corresponding corda-api change marks these interfaces as `@CordaSerializable`.
…#5107) A bug was observed whereby Kafka producer errors were not being correctly reported back to the MultiSourceEventMediator. This is because our Kafka messages are essentially 'fire-and-forget'; when something goes wrong, Kafka will exceptionally close a future in the background, but we never attempted to capture or track this future. This PR adds a callback which will log this exception clearly for the user.
corda-jenkins-ci02
bot
requested review from
josephzunigadaly,
mcgovc,
ben-millar,
YashNabar,
lankydan,
yift-r3,
conalsmith-r3,
emilybowe and
jujoramos
corda-jenkins-ci02
bot
requested review from
kyriathar,
ronanbrowne,
mbrkic-r3,
JamesHR3 and
driessamyn
|
Please remember to 'Merge' all forward merges and do not 'Squash and Merge' |
|
|
This PR is stale because it has been open 7 days with no activity. Remove the |
|
This PR was closed because it has been stale for 7 days with no activity. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR was created by the merge bot.
Files with conflicts to resolve manually:
gradle.propertiesIncludes:
findUnconsumedStatesByExactType(CORE-18370 FixfindUnconsumedStatesByExactType#5111)fetch.max.wait.msdefault config (CORE-18194 - Revertfetch.max.wait.msdefault config #5050)