CORE-18015: Merging forward updates from release/os/5.1 to release/interop/syntax - 2023-10-26 #4992
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![corda-jenkins-ci02[bot]](https://avatars.githubusercontent.com/u/22600631?s=60&v=4){kind=small}
Move the Token Selection Processor in the stand-alone worker.
This reverts commit e2aab1c.
Fix update and delete SQL statements within the State Manager implementation so only the states with the specified key and version are updated. This prevents race conditions while executing the optimistic locking and also reduces by half the amount of SQL statements that needs be executed for every update and delete.
This commit provides the ability to override topic configuration, including replica and partition counts, on the topic create CLI commands and via the Helm chart. The YAML file output by the topic create preview now includes partitions and replicas fields per topic. These default to the global parameters. These are then consumed by the topic create connect command so can be manually modified between preview and use. The topic create preview and topic create connect commands also both take an --overides/-o option, which is the optional path to a file of the same format as the preview file. This file's contents are merged to override those generated/passed in. The contents of this file can be provided as YAML under bootstrap.kafka.overrides.
#4827) Fixes exception handling for minimum platform version submitted using the MGM API to improve the returned error message.
CORE-17431 Store metadata in utxo_transaction_metadata since it is relatively large but only changes occasionally.
From logs locally and in our resiliency tests it's clear sandbox closing and therefore bundle uninstallation can be performed in multiple threads depending on whether purgeExpiryQueue is called from the cache's executor thread via the removal listener, or as part of some other synchronous cache operation which happens in the thread of the client calling it. The parallelism in SecManagerForkJoinPool is configured to the number of processors. This means we wont just see the members of SandboxGroupContextCacheImpl manipulated across 2 threads, we will see it across many and it could happen concurrently in periods of significant flow activity. This change makes the sandbox cache single threaded, meaning we only ever see manipulation of SandboxGroupContextCacheImpl on the same thread - confirmed by examining the logs. The author of Caffeine states here that using the same thread for the executor is quite acceptable, the Cache itself does little work there: https://stackoverflow.com/questions/68101454/caffeine-combining-both-scheduler-and-executor-service
…' in ExpirationProcessor (#4839) We added a new config value to define the polling frequency for the ExpirationProcessor. Previously, the processor failed to reach a healthy lifecycle state when we did a platform upgrade, because there was no mechanism to push the newly introduced config values to Kafka. We added a temporary fix which would set the value of the frequencyOfExpirationPoll to a default in case of a missing config exception happens on upgrade. We now have in place a fix to force initial config reconciliation on platform upgrade scenarios, so this temporary fix should be safe to remove.
Update sandbox level cache to receive virtualNodeContext to fix CPI upgrade failure
…sting endpoint (#4774) For the RPC Epic the Persistence api will require two endpoints. One endpoint is already merged which relates to ledger-related persistence. This PR will be making a suggested change to the naming of that endpoint for clarity, and will also add the second endpoint required. When this work is merged we will have the endpoints: /ledger - for ledger persistence /persistence - for entity persistence
…flow-changes-revert
…le and align with column changes (#4849) The utxo_transaction_sources table has been modified. See CORE-17430 Modify the utxo_transaction_sources table by removing unused columns and renaming corda-api#1288. This PR contains the changes to align with those DB changes. Reference states were not persisted in utxo_transaction_sources until now. Added that persisting logic.
This moves the RPC worker endpoint paths (E.g. `/ledger`) to a common consts file so they can be used to define both client and server config.
1. Do not rely on order of the requests returned. 2. Add more logging when re-registration is actually happening.
…ated state manager db (#4763) These changes enable an isolated state-manager database to be automatically deployed via the Corda helm chart when db bootstrapping is enabled. A state-manager database can be bootstrapped by providing the `bootstrap.db.stateManager` postgres credentials overrides, and a `stateManager.db.host` override, along with credentials in `stateManager.db`. If no overrides are provided, the state-manager will be bootstrapped in a schema in the cluster database with cluster credentials. Template functions in _helpers.tpl will fall back to cluster JDBC URL and credentials.
CORE-16203, CORE-17429: Flow changes revert
Remove the attachment API from UTXO ledger transactions.
#4863) Original scope was: The below conversation shows that if certificate validation fails, it’s unclear which certificate the error refers to. In this case it must refer to the TLS certificate because the session certificate is not used, but the error message should be clearer regardless. However, we lately merged in a fix where we specify which type of certificate was wrong, so the message should be okay. This change resulted in Internal Server Error 500, which is wrong so modified it to 400.
This PR adds a `RPCClient` which will, in the future, be used by the Multi-source Event Mediator to make calls to Corda workers via RPC. Also included is a simple `HTTPRetryExecutor` which will retry HTTP calls a configurable number of times with incremental backoff and allows for a user-specified set of retryable exceptions.
Add `findUnconsumedStatesByExactType` to `UtxoLedgerService`, allowing developers to retrieve states by type from the vault in pages. This dissuades developers from retrieving every state from the vault in one go and returning them to a flow. Reuse the vault-named query logic to create a `findUnconsumedStatesByExactType` API that supports paging without too much extra work. Remove the non-paged version of `findUnconsumedStatesByExactType` as well.
Adds support for state metadata to the state and event processor. Components using the event mediator to process events and states may wish to set metadata alongside the state that is written back to the underlying state storage. This change adds to the StateAndEventProcessor API to facilitate this, as well as migrating existing state and event processors to use this. --------- Co-authored-by: James Higgs <[email protected]>
corda-jenkins-ci02
bot
requested review from
nikinagy,
LWogan,
conalsmith-r3,
driessamyn,
mbrkic-r3,
dickon,
JamesHR3,
ben-millar,
davidcurrie,
ronanbrowne,
filipesoliveira,
josephzunigadaly,
jujoramos,
vkolomeyko,
Tom-Fitzpatrick,
lankydan,
wzur-r3,
jacob-scott3141,
yift-r3,
williamvigorr3,
YashNabar,
relyafi,
blsemo,
nkovacsx,
emilybowe,
jennyang-r3,
simon-johnson-r3 and
vlajos
ac101m
deleted the
merge-release/os/5.1-release/interop/syntax-2023-10-26-215
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR was created by the merge bot.
Files with conflicts to resolve manually:
applications/workers/workers-smoketest/src/smokeTest/kotlin/net/corda/applications/workers/smoketest/flow/FlowTests.ktcomponents/flow/flow-mapper-impl/src/main/kotlin/net/corda/flow/mapper/impl/RecordFactoryImpl.ktcomponents/flow/flow-mapper-impl/src/test/kotlin/net/corda/flow/mapper/impl/executor/SessionInitProcessorTest.ktcomponents/flow/flow-service/src/main/kotlin/net/corda/flow/service/FlowService.ktgradle.propertieslibs/membership/membership-impl/src/main/kotlin/net/corda/membership/lib/impl/grouppolicy/GroupPolicyParserImpl.ktIncludes:
VirtualNodeWriteEventHandler(CORE-17867 - Added logging toVirtualNodeWriteEventHandler#4959)UtxoSignedLedgerTransactionKryoSerializer(CORE-17837 AddUtxoSignedLedgerTransactionKryoSerializer#4945)findUnconsumedStatesByType(CORE-17782 Suppress deprecation warning onfindUnconsumedStatesByType#4884)findUnconsumedStatesByExactTypepaging API (CORE-15552 AddfindUnconsumedStatesByExactTypepaging API #4837)