Ensure lifecycle tasks wait for messages to be pushed

[francoisferrand]

Issue: BB-641

[bert-e]

Hello francoisferrand,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[codecov]

Codecov Report

Attention: Patch coverage is 84.37500% with 5 lines in your changes missing coverage. Please review.

Project coverage is 55.34%. Comparing base (af16d44) to head (ca50af2).

Files with missing lines	Patch %	Lines
extensions/lifecycle/tasks/LifecycleTaskV2.js	87.50%	3 Missing ⚠️
extensions/lifecycle/tasks/LifecycleTask.js	75.00%	2 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
extensions/lifecycle/tasks/LifecycleTask.js	83.27% <75.00%> (+0.08%)	⬆️
extensions/lifecycle/tasks/LifecycleTaskV2.js	89.56% <87.50%> (+0.67%)	⬆️

... and 1 file with indirect coverage changes

Components	Coverage Δ
Bucket Notification	18.51% <ø> (ø)
Core Library	61.92% <ø> (+0.01%)	⬆️
Ingestion	67.53% <ø> (ø)
Lifecycle	47.09% <84.37%> (+0.18%)	⬆️
Oplog Populator	84.20% <ø> (ø)
Replication	51.01% <ø> (ø)
Bucket Scanner	85.60% <ø> (ø)

@@                 Coverage Diff                 @@
##           development/8.6    #2603      +/-   ##
===================================================
+ Coverage            55.30%   55.34%   +0.04%     
===================================================
  Files                  198      198              
  Lines                12915    12925      +10     
===================================================
+ Hits                  7142     7153      +11     
+ Misses                5763     5762       -1     
  Partials                10       10              

Flag	Coverage Δ
api:retry	9.62% <0.00%> (-0.01%)	⬇️
api:routes	9.51% <0.00%> (-0.01%)	⬇️
bucket-scanner	85.60% <ø> (ø)
ingestion	12.46% <0.00%> (-0.01%)	⬇️
lib	7.52% <0.00%> (+<0.01%)	⬆️
lifecycle	19.43% <84.37%> (+0.08%)	⬆️
notification	0.88% <0.00%> (-0.01%)	⬇️
replication	18.87% <0.00%> (-0.02%)	⬇️
unit	5.13% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[bert-e]

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

[williamlardier]

With this code, in case of error in _compareRulesToList, we are ignoring the returned error, as we would have it in the "then" with the rejected status. This looks like a behavior change we don't want? We ignore errors for _sendBucketEntry but for _compareRulesToList, we actually call the reject function...

Promise.allSettled will return an array with the status for each promise, including the ones rejected:

Promise.allSettled([
  Promise.resolve(33),
  new Promise((resolve) => setTimeout(() => resolve(66), 0)),
  99,
  Promise.reject(new Error("an error")),
]).then((values) => console.log(values), err => console.log('catch', err));

[
    {
        "status": "fulfilled",
        "value": 33
    },
    {
        "status": "fulfilled",
        "value": 66
    },
    {
        "status": "fulfilled",
        "value": 99
    },
    {
        "status": "rejected",
        "reason": {}
    }
]

(and no "catch" log).

And if we are aligned, it also means this is something we do not test, or returning the error in the callback has no impact...

[francoisferrand]

you are right (and I will update the code), but actually _compareRulesToList is not upposed to fail either : it calls _retryEntry, which ignores the errors. Errors on individual objects are retried a bit ; but eventually we just skip the object (or batch), relying on the periodic scan to eventually process these obejcts.