Collect requirements for v2.5.0

[yanji09]

Harbor team is planning for v2.5.0

If any features you would like to see in the next release, please input your ideas/requirements in this discussion.
If there's already an issue opened for a feature, please refer it here as well.

Thanks.

[Vad1mo]

I would like 2.5 to focus more on stability and resilience:
A perhaps somewhat wild claim from my side is that Harbor can't deal with storage sizes beyond > 1-2 TB. Especially on object storage.

1. GC Speedup and resilience.
   GC should be more resilient for flaky backends #15469
   GC fails when manifest not found #15822
   GC fails after deleting many tags from same repository #15807
   [GC performance] The performance of v2 manifest deletion is not good in S3 environment #12948
   GC fails with "invalid checksum digest format" from registry when deleting manifests #15970

2. Replication resilience
   I will lookup and add the open issues in that regard as a reference here.

[dkulchinsky]

Thank you for bringing this up @Vad1mo! I'd like to chime in and share our mutual concern, as our usage of Harbor grows we encounter perfromance and stability constraints that are concerning, in both of our environments (test & prod) GC has become inoperable due to the issues mentioned above.

I would love to see some focus to address these issues.

[juliohm1978]

@Vad1mo

1-2 TB seems like a small threshold to start having problems. That's actually pretty concerning, IMO. Can you ellaborate on any problems you already had?

[dkulchinsky]

@juliohm1978, indeed it's a concern - if you take a look at the issues @Vad1mo linked I think you'll see what kind of problems we're hitting, in our experience it's mostly GC related, but we're also starting to see issues around retention policies execution - taking a long time and consuming considerable resources, for reference:

• Tag deletion performance is very poor with a high number of tags #14708

[dkulchinsky]

@Vad1mo, please also add:

• GC fails with "invalid checksum digest format" from registry when deleting manifests #15970

to the GCs issue list above

[dkulchinsky]

I would also like to raise the feature request:

• Feature Request: allow selecting projects and repos to be scanned by IMAGE_SCAN_ALL #15652

as our environments grow, we feel we need better control on how scheduled image scan-all are ran, specifically we would like the ability to filter our projects and/or repositories from these scans, for example: we have >20K artifacts that are image layer caches managed by our CI build system and scanning them is basically a waste of time and resources.

In addition, scanning images in a large environment can take up to 24 hours and more and consumes considerable resources, especially available jobservice workers, which "blocks" scans that are initiated by image pushes and require a scan result during CI to validate the new image being pushed, since scan-all basically consumes all available worker slots these scan-on-push jobs can be in the queue for many minutes, even hours.

so perhaps a more granular management of the workers utilization? for example that the scan-all job can't take more than X% of the overall available workers?

These are issues we're stumbling as our environment grow and I think worth addressing.

[Vad1mo]

Here is the summary of this discussion as an issue.

#16010

[sixeela]

Implement a modulable permission system (custom groups/custom permissions per user)