*DO NOT MERGE* Topology changes - One consumer for multiple topics #4955
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Adds support for state metadata to the state and event processor. Components using the event mediator to process events and states may wish to set metadata alongside the state that is written back to the underlying state storage. This change adds to the StateAndEventProcessor API to facilitate this, as well as migrating existing state and event processors to use this. --------- Co-authored-by: James Higgs <[email protected]>
…diator in FlowWorker (#4832) Using Multi-Source Event Mediator instead of State and Event Subscription in FlowWorker. Polling consumers and committing offsets performed in mediator thread in a synchronous way (without using coroutines)
…saging-defaults.conf
Add support for parallel flows executing within the flow pipeline. This includes: - Temporary multi-thread implementation of the state and event pattern. - Using the SameThreadExecutor for flow fibers, which causes the fiber to run on the thread that started the fiber. - Using a Kryo Pool for Kryo instances, as kryo instances are not thread-safe themselves. Add TaskManager and a TaskManagerFactory which is a wrapper around a ScheduledExecutorService used for gathering metrics on executing tasks. The built-in Micrometer metrics for executor services are used, along with some additional metrics in the TaskManager that are more specific to our use-cases. --------- Co-authored-by: Conal Smith <[email protected]> Co-authored-by: Dan Newton <[email protected]> Co-authored-by: Dries Samyn <[email protected]>
…integration with StateManager (#4874) Integration of Multi-Source Event Mediator with StateManager.
…tion CORE-15419 5.1 Performace Integration
…e Event Mediator (#4875) Using Multi-Source Event Mediator instead of State and Event Subscription in FlowMapperWorker. --------- Co-authored-by: James Higgs <[email protected]>
…tion-2 CORE-15419 5.1 Performance Integration #2
… flow.mapper.start, flow.mapper.session.in, flow.mapper.session.out).
…topoligy-new-topics
…mediator-metrics-2
…rics-2 CORE-17562 Metrics for Multi-Source Event Mediator
…rker (#4820) HTTP RPC server integration into the Crypto processor. Allows Crypto processor to receive HTTP external event requests sent from the flow engine. Client integration will come later.
Adds the state status key to flow mapper state metadata and populates it with the current state. The metadata key is used by the mapper cleanup logic to ensure that mapper states are removed.
… mediator for flow event processing.
…point (#4889) Adds session timeout to the metadata stored alongside the checkpoint. Session timeout is implemented using the state storage lookup mechanism, which requires the metadata to be populated with the session timeout key. The earliest timestamp of session expiry for currently open sessions is the value assigned to this property.
…PC (#4885) This PR adds a new implementation of MessagingClient, RPCClient, handling synchronous calls to the Corda workers via RPC/HTTP with configurable retries. Includes also a small bug-fix for the task manager. --------- Co-authored-by: Miljenko Brkic <[email protected]>
…topoligy-new-topics
…w-topics' into mbrkic-r3/CORE-17768/topoligy-new-topics
…o mbrkic-r3/CORE-17768/topoligy-new-topics # Conflicts: # components/flow/flow-service/src/main/kotlin/net/corda/flow/messaging/mediator/FlowEventMediatorFactoryImpl.kt
…o mbrkic-r3/CORE-17768/topoligy-new-topics # Conflicts: # components/flow/flow-mapper-service/src/integrationTest/kotlin/net/corda/session/mapper/service/integration/TestStateManagerFactoryImpl.kt
…erviceIntegrationTest-fix-backup' into mbrkic-r3/CORE-17768/topoligy-new-topics # Conflicts: # components/flow/flow-mapper-service/src/integrationTest/kotlin/net/corda/session/mapper/service/integration/FlowMapperServiceIntegrationTest.kt # components/flow/flow-mapper-service/src/integrationTest/kotlin/net/corda/session/mapper/service/integration/TestFlowEventMediatorFactoryImpl.kt # libs/messaging/messaging-impl/src/main/kotlin/net/corda/messaging/mediator/MultiSourceEventMediatorImpl.kt
This reverts commit d78533b.
…ORE-17768/topoligy-new-topics # Conflicts: # components/flow/flow-mapper-service/src/integrationTest/kotlin/net/corda/session/mapper/service/integration/FlowMapperServiceIntegrationTest.kt # components/flow/flow-mapper-service/src/integrationTest/kotlin/net/corda/session/mapper/service/integration/TestFlowEventMediatorFactoryImpl.kt # components/flow/flow-service/src/main/kotlin/net/corda/flow/messaging/mediator/FlowEventMediatorFactoryImpl.kt # components/flow/flow-service/src/test/kotlin/net/corda/flow/messaging/FlowEventMediatorFactoryImplTest.kt
…"flow.event" topic
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
mbrkic-r3
changed the base branch from
mbrkic-r3/CORE-17768/topoligy-new-topics
to
feature/51-perf-integration
|
Jenkins build for PR 4955 build 7 Build Successful: |
…ORE-17768/topoligy-new-topics-1-consumer # Conflicts: # components/flow/flow-mapper-service/src/integrationTest/kotlin/net/corda/session/mapper/service/integration/TestFlowEventMediatorFactoryImpl.kt # components/flow/flow-mapper-service/src/main/kotlin/net/corda/session/mapper/messaging/mediator/FlowMapperEventMediatorFactoryImpl.kt # components/flow/flow-service/src/main/kotlin/net/corda/flow/messaging/mediator/FlowEventMediatorFactoryImpl.kt
mbrkic-r3
changed the base branch from
feature/51-perf-integration
to
release/os/5.1
|
This PR is stale because it has been open 7 days with no activity. Remove the |
|
This PR was closed because it has been stale for 7 days with no activity. |
github-actions
bot
deleted the
mbrkic-r3/CORE-17768/topoligy-new-topics-1-consumer
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is alternative solution to #4931
Using new topology Kafka topics (flow.start, flow.session, flow.mapper.start, flow.mapper.session.in, flow.mapper.session.out).
One Kafka consumer subscribes to multiple input topics.