CORE-18532: Retry Transient Errors in Flow Engine

[jujoramos]

Instead of automatically publishing the original event back to the
"flow.event" topic whenever a transient exception occurs when executing
the flow pipeline, automatically retry the exceptions using an
exponential backoff retry mechanism and permanently fail the flow if
the configured time or attemps is reached.

• Create internal retry utility to manage retries in an automated
  fashion with plugable backoff strategy (constant, linear and
  exponential provided out of the box).
• Replace existing utilities in the crypto components and tests with
  the new one and delete unused classes.
• Update FlowEventProcessor to automatically retry transient exceptions
  through the new retry utility using an exponential backoff with a
  growth factor of 250ms.
• Remove unnecessary code and update tests to accommodate for the new
  internal retry.

[corda-jenkins-ci02]

Jenkins build for PR 5337 build 21

Build Successful:
Jar artifact version produced by this PR: 5.2.0.0-alpha-1704304261991
Helm chart version produced by this PR: 5.2.0-alpha.1704304261991
Helm chart pushed to: oci://corda-os-docker-dev.software.r3.com/helm-charts/pr-5337/corda

[corda-jenkins-ci02]

Building E2E Tests on PR-5337

[dickon]

I added a bunch of comments, which need not blocking merging so if you want to disregard them I can approve anyway.

[kyriathar]

Some minor comments. The crypto bits look good to me in principle.

[driessamyn]

Did you consider using an off-the-shelf library like Resilience4J rather than rolling our own? (not saying we should, but we should evaluate pros/cons).

Also, I thought we'd look at moving away from crypto doing its own bespoke thing and consolidate how we do retries etc with the RPC pattern?

[jujoramos]

Did you consider using an off-the-shelf library like Resilience4J rather than rolling our own? (not saying we should, but we should evaluate pros/cons).

I did at the beginning but our use case is quite simplistic at the moment, adding resilience4j felt like killing mosquitos with a cannon... not the most technically valid reason in the world 🤷‍♂️ .

Also, I thought we'd look at moving away from crypto doing its own bespoke thing and consolidate how we do retries etc with the RPC pattern?

Not sure what the background context is here, I've simply created a retry utility to be shared across the code base and refactored the existing crypto retry mechanism to use the new one. Any other work to replace by RPC pattern should likely be its own epic/task, not included within this PR.

[conalsmith-r3]

Also, I thought we'd look at moving away from crypto doing its own bespoke thing and consolidate how we do retries etc with the RPC pattern?

Yeah I don't remember us discussing or agreeing anything. But as I'm working on handling transient errors for external processors the crypto retry logic breaks the mould. If @dickon agrees we could add a story to remove crypto retry handling and fall back on the RPC client's retry logic, the epic is here.

[kyriathar]

OK (for crypto)

[LWogan]

These changes look fine to me.

[sonarqubecloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

[dickon]

LGTM

[LWogan]

LGTM