CORE-17768 Topology changes - Use new topics #4931
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds support for state metadata to the state and event processor. Components using the event mediator to process events and states may wish to set metadata alongside the state that is written back to the underlying state storage. This change adds to the StateAndEventProcessor API to facilitate this, as well as migrating existing state and event processors to use this. --------- Co-authored-by: James Higgs <[email protected]>
…diator in FlowWorker (#4832) Using Multi-Source Event Mediator instead of State and Event Subscription in FlowWorker. Polling consumers and committing offsets performed in mediator thread in a synchronous way (without using coroutines)
…saging-defaults.conf
Add support for parallel flows executing within the flow pipeline. This includes: - Temporary multi-thread implementation of the state and event pattern. - Using the SameThreadExecutor for flow fibers, which causes the fiber to run on the thread that started the fiber. - Using a Kryo Pool for Kryo instances, as kryo instances are not thread-safe themselves. Add TaskManager and a TaskManagerFactory which is a wrapper around a ScheduledExecutorService used for gathering metrics on executing tasks. The built-in Micrometer metrics for executor services are used, along with some additional metrics in the TaskManager that are more specific to our use-cases. --------- Co-authored-by: Conal Smith <[email protected]> Co-authored-by: Dan Newton <[email protected]> Co-authored-by: Dries Samyn <[email protected]>
…integration with StateManager (#4874) Integration of Multi-Source Event Mediator with StateManager.
…tion CORE-15419 5.1 Performace Integration
…e Event Mediator (#4875) Using Multi-Source Event Mediator instead of State and Event Subscription in FlowMapperWorker. --------- Co-authored-by: James Higgs <[email protected]>
…tion-2 CORE-15419 5.1 Performance Integration #2
… flow.mapper.start, flow.mapper.session.in, flow.mapper.session.out).
…topoligy-new-topics
…mediator-metrics-2
…rics-2 CORE-17562 Metrics for Multi-Source Event Mediator
…rker (#4820) HTTP RPC server integration into the Crypto processor. Allows Crypto processor to receive HTTP external event requests sent from the flow engine. Client integration will come later.
Adds the state status key to flow mapper state metadata and populates it with the current state. The metadata key is used by the mapper cleanup logic to ensure that mapper states are removed.
… mediator for flow event processing.
…point (#4889) Adds session timeout to the metadata stored alongside the checkpoint. Session timeout is implemented using the state storage lookup mechanism, which requires the metadata to be populated with the session timeout key. The earliest timestamp of session expiry for currently open sessions is the value assigned to this property.
…PC (#4885) This PR adds a new implementation of MessagingClient, RPCClient, handling synchronous calls to the Corda workers via RPC/HTTP with configurable retries. Includes also a small bug-fix for the task manager. --------- Co-authored-by: Miljenko Brkic <[email protected]>
…topoligy-new-topics
…w-topics' into mbrkic-r3/CORE-17768/topoligy-new-topics
…o mbrkic-r3/CORE-17768/topoligy-new-topics # Conflicts: # components/flow/flow-service/src/main/kotlin/net/corda/flow/messaging/mediator/FlowEventMediatorFactoryImpl.kt
mbrkic-r3
requested review from
rick-r3,
simon-johnson-r3,
blsemo,
Omar-awad,
aditisdesai,
vinir3,
vkolomeyko,
thiagoviana and
Sakpal
as code owners
This was referenced Oct 24, 2023
lankydan
approved these changes
Oct 24, 2023
…RE-17768/topology-changes-kafka-config
…RE-17768/topology-changes-kafka-config
mbrkic-r3
deleted the
mbrkicr3/CORE-17768/topology-changes-kafka-config
branch
vkolomeyko
added a commit
that referenced
this pull request
Oct 26, 2023
This reverts commit c706f7c.
Omar-awad
added a commit
that referenced
this pull request
Oct 27, 2023
CORE-17768: Revert - Topology changes - Use new topics (#4931)
mbrkic-r3
added a commit
that referenced
this pull request
Oct 27, 2023
New topology Kafka topics are being used (flow.start, flow.session, flow.mapper.start, flow.mapper.session.in, flow.mapper.session.out).
Consumers are polled sequentially and performance impact (related to topics with low traffic) is mitigated by adjusting configuration:
- fetch.max.wait.ms = 20
- pollTimeout = 20ms
- max.poll.records = 100
- linger.ms = 50
- batch.size = 204800
- partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
- group.id=${group}-cooperative
Missing topic permissions are set (CORE-17768 Topology changes - Use new topics corda-api#1306)
---------
Co-authored-by: James Higgs <[email protected]>
Co-authored-by: Conal Smith <[email protected]>
Co-authored-by: Dan Newton <[email protected]>
Co-authored-by: Dries Samyn <[email protected]>
Co-authored-by: Omar Awad <[email protected]>
Co-authored-by: Thiago Viana <[email protected]>
Co-authored-by: David Currie <[email protected]>
Co-authored-by: James Higgs <[email protected]>
Co-authored-by: Ben Millar <[email protected]>
Omar-awad
added a commit
that referenced
this pull request
Nov 1, 2023
…-topics-omar CORE-17768 Topology changes - Use new topics (#4931)
mbrkic-r3
restored the
mbrkicr3/CORE-17768/topology-changes-kafka-config
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With this change:
New topology Kafka topics are being used (
flow.start,flow.session,flow.mapper.start,flow.mapper.session.in,flow.mapper.session.out).Consumers are polled sequentially and performance impact (related to topics with low traffic) is mitigated by adjusting configuration:
fetch.max.wait.ms = 20pollTimeout = 20msmax.poll.records = 100linger.ms = 50batch.size = 204800partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignorgroup.id=${group}-cooperativeMissing topic permissions are set (CORE-17768 Topology changes - Use new topics corda-api#1306)