Improvement/s3 utils 163

[williamlardier]

This PR fixes a limitation we have: where the in-flights would be dropped if coming after the associated bucket was scanned.

We chose to recompute them by:

• First getting the in-flight when starting the scan.
• Then getting the in-flights of all buckets at the end of the script. We then reapply the diff between the end in-flights minus the start in-flights in _inflightDelta, and included in Current to be accounted for by Scuba.
• Finally, we pass the account owner ids in the bucket documents, so that we can sum all the in-flights from buckets and store it in the associated account.

Note: a known limitation comes from the fact that if there is only quota on account, we cannot track the quotas. This is something to be fixed in CLDSRV in a 2nd iteration, by always updating both account and bucket in-flights, even if there is only one quota, and with a single scubaclient call.

---

Example of output:

{
	"_id" : "account_79ed68e510217795c4df855b2041fee0a7cf1c35565a4ea50639cb0644cf2aff",
	"measuredOn" : "2024-05-17T16:08:04.113Z",
	"usedCapacity" : {
		"current" : NumberLong(4501559),
		"nonCurrent" : NumberLong(186900),
		"_currentCold" : NumberLong(0),
		"_nonCurrentCold" : NumberLong(0),
		"_currentRestored" : NumberLong(0),
		"_currentRestoring" : NumberLong(0),
		"_nonCurrentRestored" : NumberLong(0),
		"_nonCurrentRestoring" : NumberLong(0),
		"_inflightsPreScan" : NumberLong(22),
		"_inflightsDelta" : NumberLong(22)
	},
	"objectCount" : {
		"current" : NumberLong(1489660),
		"nonCurrent" : NumberLong(1869),
		"_currentCold" : NumberLong(0),
		"_nonCurrentCold" : NumberLong(0),
		"_currentRestored" : NumberLong(0),
		"_currentRestoring" : NumberLong(0),
		"_nonCurrentRestored" : NumberLong(0),
		"_nonCurrentRestoring" : NumberLong(0),
		"deleteMarker" : NumberLong(0)
	},
	"locations" : {
		"a43f28f9-135f-11ef-b984-faa7d2beb2b5" : {
			"usedCapacity" : {
				"current" : NumberLong(4501537),
				"nonCurrent" : NumberLong(186900),
				"_currentCold" : NumberLong(0),
				"_nonCurrentCold" : NumberLong(0),
				"_currentRestored" : NumberLong(0),
				"_currentRestoring" : NumberLong(0),
				"_nonCurrentRestored" : NumberLong(0),
				"_nonCurrentRestoring" : NumberLong(0),
				"_inflightsPreScan" : NumberLong(0)
			},
			"objectCount" : {
				"current" : NumberLong(1489660),
				"nonCurrent" : NumberLong(1869),
				"_currentCold" : NumberLong(0),
				"_nonCurrentCold" : NumberLong(0),
				"_currentRestored" : NumberLong(0),
				"_currentRestoring" : NumberLong(0),
				"_nonCurrentRestored" : NumberLong(0),
				"_nonCurrentRestoring" : NumberLong(0),
				"deleteMarker" : NumberLong(0)
			}
		}
	}
}

{
	"_id" : "bucket_bucket1_1715849127256",
	"measuredOn" : "2024-05-17T16:08:04.113Z",
	"usedCapacity" : {
		"current" : NumberLong(2387059),
		"nonCurrent" : NumberLong(0),
		"_currentCold" : NumberLong(0),
		"_nonCurrentCold" : NumberLong(0),
		"_currentRestored" : NumberLong(0),
		"_currentRestoring" : NumberLong(0),
		"_nonCurrentRestored" : NumberLong(0),
		"_nonCurrentRestoring" : NumberLong(0),
		"_inflightsDelta" : NumberLong(22),
		"_inflight" : 22
	},
	"objectCount" : {
		"current" : NumberLong(1468515),
		"nonCurrent" : NumberLong(0),
		"_currentCold" : NumberLong(0),
		"_nonCurrentCold" : NumberLong(0),
		"_currentRestored" : NumberLong(0),
		"_currentRestoring" : NumberLong(0),
		"_nonCurrentRestored" : NumberLong(0),
		"_nonCurrentRestoring" : NumberLong(0),
		"deleteMarker" : NumberLong(0)
	}
}

[bert-e]

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[williamlardier]

/approve

[benzekrimaha]

LGTM ( modulo the tests that were skipped probably a leftover ? )

[bert-e]

Integration data created

I have created the integration data for the additional destination branches.

• this pull request will merge improvement/S3UTILS-163 into
  development/1.14
• w/1.15/improvement/S3UTILS-163 will be merged into development/1.15

The following branches will NOT be impacted:

• development/1.13
• development/1.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

[bert-e]

I have successfully merged the changeset of this pull request
into targetted development branches:

• ✔️ development/1.14

• ✔️ development/1.15

The following branches have NOT changed:

• development/1.13
• development/1.4

Please check the status of the associated issue S3UTILS-163.

Goodbye williamlardier.

The following options are set: approve