add retries to hotswapping resources for sandbox tests

[rtpascual]

Problem

Sometimes hotswapping resources takes a few seconds longer than our threshold so E2E throws with:

test at packages/integration-tests/lib/test-e2e/sandbox/sandbox.test.template.js:53:22
✖ [data-storage-auth] hot-swaps a change (107752.657625ms)
  Error: Deployment time 33.6 seconds is higher than the threshold of 30
      at Object.action (/Users/runner/work/amplify-backend/amplify-backend/packages/integration-tests/src/process-controller/predicated_action_queue_builder.ts:106:17)
      at ProcessController.run (/Users/runner/work/amplify-backend/amplify-backend/packages/integration-tests/src/process-controller/process_controller.ts:91:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async TestContext.<anonymous> (/Users/runner/work/amplify-backend/amplify-backend/packages/integration-tests/src/test-e2e/sandbox/sandbox.test.template.ts:106:13)
      at async Test.run (node:internal/test_runner/test:797:9)
      at async Suite.processPendingSubtests (node:internal/test_runner/test:527:7)

Issue number, if available:

Changes

Add retries if hotswap deployment time is higher than the set threshold.

Corresponding docs PR, if applicable:

Validation

E2E tests. Also verified locally that test tries to hotswap 3 times when setting threshold to 1 second.

Checklist

• If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
• If this PR requires a change to the Project Architecture README, I have included that update in this PR.
• If this PR requires a docs update, I have linked to that docs PR above.
• If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[changeset-bot]

🦋 Changeset detected

Latest commit: c07dc95

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[sobolk]

could you add a code comment explaining that it's important to keep initial deployment in retry loop ?

I.e. the app state must be reset in each retry for updates (hotswaps) to be non no-op.