Improvement/s3 utils 155 granular metrics #309

williamlardier · 2024-03-25T16:58:13Z

Update the job to:

Add granularity for objects in cold storage, being restored, or restored, in all the computed metrics. Code architecture is kept, this only adds linear complexity (that we have no choice but to compute).
Add more unit and functionsl tests to cover cold objects (restoring, restored and in cold).
Change the logic when an object is being restored, to count it also in the destination location (i.e., the hot one). This ensures the returned metric is consistent with the upcoming quota checks: we consider objects pending restoration as already part of the metrics of the destination (hot) location. This only applies to locations.

Main commit to review : 4317c3e

bert-e · 2024-03-25T16:58:16Z

Hello williamlardier,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2024-03-25T16:58:20Z

Incorrect fix version

The Fix Version/s in issue S3UTILS-155 contains:

None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

1.14.5
1.15.0

Please check the Fix Version/s of S3UTILS-155, or the target
branch of this pull request.

williamlardier · 2024-03-26T15:33:24Z

FYI @KazToozs @BourgoisMickael , I added a commit that also splits the restoring and restored into current & nonCurrent. It makes more sense in terms of logic, and it also ensures backward compatibility with the UI (where Pensieve-API will return the right aggregated metric transparently).

Edit: added another one to keep the original values for current/nonCurrent, making the code fully backward compatible, without altering existing projects, and ability to know the data not in cold.

williamlardier · 2024-03-26T15:42:26Z

/create_integration_branches

bert-e · 2024-03-26T15:42:35Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/S3UTILS-155-granular-metrics into
development/1.14
w/1.15/improvement/S3UTILS-155-granular-metrics will be merged into development/1.15

The following branches will NOT be impacted:

development/1.13
development/1.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-03-26T15:42:36Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

williamlardier · 2024-03-26T16:13:15Z

/reset

bert-e · 2024-03-26T16:13:23Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2024-03-26T16:13:30Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/S3UTILS-155-granular-metrics into
development/1.14
w/1.15/improvement/S3UTILS-155-granular-metrics will be merged into development/1.15

The following branches will NOT be impacted:

development/1.13
development/1.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-03-26T16:13:31Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

We introduce new fields: - (non)CurrentCold fields are counting the objects that have archive set in their metadata, i.e., in cold storage. - restoring tracks objects being restored. - restored tracks objects that are restored, and not yet expired.

…rent

Thsi ensures backward compatibility and remove the need to update the services using theses values: ZKUI and PENSIVE-API

williamlardier · 2024-04-10T07:56:31Z

/reset

bert-e · 2024-04-10T07:56:38Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2024-04-10T07:56:44Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/S3UTILS-155-granular-metrics into
development/1.14
w/1.15/improvement/S3UTILS-155-granular-metrics will be merged into development/1.15

The following branches will NOT be impacted:

development/1.13
development/1.4

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2024-04-10T07:56:45Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

williamlardier · 2024-04-10T08:12:00Z

/approve

bert-e · 2024-04-10T08:12:13Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/1.14
✔️ development/1.15

The following branches have NOT changed:

development/1.13
development/1.4

Please check the status of the associated issue S3UTILS-155.

Goodbye williamlardier.

The following options are set: approve, create_integration_branches

williamlardier force-pushed the improvement/S3UTILS-155-granular-metrics branch from 86f3348 to 5596b25 Compare March 25, 2024 17:04

BourgoisMickael approved these changes Mar 26, 2024

View reviewed changes

KazToozs approved these changes Mar 26, 2024

View reviewed changes

williamlardier force-pushed the improvement/S3UTILS-155-granular-metrics branch from 6ec9995 to 461f8f3 Compare March 26, 2024 15:34

williamlardier force-pushed the improvement/S3UTILS-155-granular-metrics branch from fc3f28a to 712edf2 Compare March 26, 2024 16:12

scality deleted a comment from bert-e Mar 26, 2024

williamlardier added 10 commits April 10, 2024 09:55

S3UTILS-155: handle new fields for SOSAPI computations

47cb4e0

S3UTILS-155: compute metrics with more granularity

a228def

S3UTILS-155: adapt logic with object cold state

0f26e06

S3UTILS-155: unit tests

c51b25c

S3UTILS-155: functional tests updates and cold objects

01a142e

S3UTILS-155: update unit tests for count manager

b424a51

S3UTILS-155: split the restoring and restored into current and nonCur…

8f592ae

…rent

S3UTILS-155: keep the original values for current & nonCurrent

91d52df

Thsi ensures backward compatibility and remove the need to update the services using theses values: ZKUI and PENSIVE-API

S3UTILS-155: bump project version

9ac860c

williamlardier force-pushed the improvement/S3UTILS-155-granular-metrics branch from c112ccd to 9ac860c Compare April 10, 2024 07:56

scality deleted a comment from bert-e Apr 10, 2024

bert-e merged commit 9ac860c into development/1.14 Apr 10, 2024
14 checks passed

bert-e deleted the improvement/S3UTILS-155-granular-metrics branch April 10, 2024 08:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improvement/s3 utils 155 granular metrics #309

Improvement/s3 utils 155 granular metrics #309

williamlardier commented Mar 25, 2024 •

edited

Loading

bert-e commented Mar 25, 2024

bert-e commented Mar 25, 2024

williamlardier commented Mar 26, 2024 •

edited

Loading

williamlardier commented Mar 26, 2024

bert-e commented Mar 26, 2024

bert-e commented Mar 26, 2024

williamlardier commented Mar 26, 2024

bert-e commented Mar 26, 2024

bert-e commented Mar 26, 2024

bert-e commented Mar 26, 2024

williamlardier commented Apr 10, 2024

bert-e commented Apr 10, 2024

bert-e commented Apr 10, 2024

bert-e commented Apr 10, 2024

williamlardier commented Apr 10, 2024

bert-e commented Apr 10, 2024

Improvement/s3 utils 155 granular metrics #309

Improvement/s3 utils 155 granular metrics #309

Conversation

williamlardier commented Mar 25, 2024 • edited Loading

bert-e commented Mar 25, 2024

Hello williamlardier,

bert-e commented Mar 25, 2024

Incorrect fix version

williamlardier commented Mar 26, 2024 • edited Loading

williamlardier commented Mar 26, 2024

bert-e commented Mar 26, 2024

Integration data created

bert-e commented Mar 26, 2024

Waiting for approval

williamlardier commented Mar 26, 2024

bert-e commented Mar 26, 2024

Reset complete

bert-e commented Mar 26, 2024

Integration data created

bert-e commented Mar 26, 2024

Waiting for approval

williamlardier commented Apr 10, 2024

bert-e commented Apr 10, 2024

Reset complete

bert-e commented Apr 10, 2024

Integration data created

bert-e commented Apr 10, 2024

Waiting for approval

williamlardier commented Apr 10, 2024

bert-e commented Apr 10, 2024

williamlardier commented Mar 25, 2024 •

edited

Loading

williamlardier commented Mar 26, 2024 •

edited

Loading