Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable Kopia Maintenance Interval #8581

Merged
merged 11 commits into from
Feb 25, 2025

Conversation

kaovilai
Copy link
Member

@kaovilai kaovilai commented Jan 6, 2025

Signed-off-by: Tiger Kaovilai [email protected]

Thank you for contributing to Velero!

Please add a summary of your change

Configurable Kopia Maintenance Interval. repo-maintenance-job-configmap adds an option for fullMaintenanceInterval where fastGC (12 hours), and eagerGC (6 hours).
This configmap is loaded in assembleRepoParam that returns RepoParam which now contains the maintenance interval referenced by several repository functions.
The manager functions then call provider functions which read repoParam.FullMaintenanceInterval and add it to repoOption.
Repooption is then passed to repoService, in this case kopiaRepoService.Init
kopiaRepoService.Init calls writeInitParameters which finally set kopia p.FullCycle.Interval

Doc previews

https://velero-git-configkopiamaintinterval-kaovilais-projects.vercel.app/docs/main/repository-maintenance#full-maintenance-interval-customization
https://velero-git-configkopiamaintinterval-kaovilais-projects.vercel.app/docs/main/backup-repository-configuration/#full-maintenance-interval-customization

Does your change fix a particular issue?

Fixes #8364

Please indicate you've done the following:

@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch 2 times, most recently from 6e3d09f to 569c2ef Compare January 15, 2025 10:03
Copy link

codecov bot commented Jan 15, 2025

Codecov Report

Attention: Patch coverage is 84.00000% with 4 lines in your changes missing coverage. Please review.

Project coverage is 59.50%. Comparing base (0f81772) to head (178b6e3).
Report is 14 commits behind head on main.

Files with missing lines Patch % Lines
pkg/repository/udmrepo/kopialib/lib_repo.go 73.33% 4 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8581      +/-   ##
==========================================
+ Coverage   59.48%   59.50%   +0.01%     
==========================================
  Files         371      371              
  Lines       40191    40213      +22     
==========================================
+ Hits        23909    23929      +20     
- Misses      14789    14791       +2     
  Partials     1493     1493              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch from 569c2ef to bcc6e39 Compare January 15, 2025 13:13
@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch 3 times, most recently from 014d288 to 40af53f Compare January 15, 2025 13:54
@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch from 40af53f to 14200ad Compare January 15, 2025 15:13
@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch 2 times, most recently from cc27593 to 334ac16 Compare January 16, 2025 10:29
Copy link
Member Author

@kaovilai kaovilai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently double checking the flow of backup-repository-configmap from specifying a configmap to reading to storing into a repoOption.StorageOptions[key]

func getBackupRepositoryConfig(ctx context.Context, ctrlClient client.Client, configName, namespace, repoName, repoType string, log logrus.FieldLogger) (map[string]string, error) {
if configName == "" {
is called [here](
config, err := getBackupRepositoryConfig(ctx, r, r.backupRepoConfig, r.namespace, req.Name, req.Spec.RepositoryType, log)

which defines config := map[string]string with all the values set by user.
This map of string is then set to backupRepository.spec.RepositoryConfig and patched to cluster CR in following
func (r *BackupRepoReconciler) initializeRepo(ctx context.Context, req *velerov1api.BackupRepository, log logrus.FieldLogger) error {

This function is only called when backupRepo is in newly created as part of a backuprepo Reconcile()

This is used by

func (urp *unifiedRepoProvider) GetStoreOptions(param interface{}) (map[string]string, error) {

via
storeVar, err := funcTable.getStorageVariables(repoParam.BackupLocation, urp.repoBackend, repoParam.BackupRepo.Spec.VolumeNamespace, repoParam.BackupRepo.Spec.RepositoryConfig)

However, getStorageVariables() do not pass all the options from backupRepoConfig back.

if backupRepoConfig != nil {
if v, found := backupRepoConfig[udmrepo.StoreOptionCacheLimit]; found {
result[udmrepo.StoreOptionCacheLimit] = v
}
}

So either make getStorageVariables() pass everything, or make it add the maintenance interval to one of the things it adds.

@kaovilai

This comment was marked as resolved.

@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch 3 times, most recently from 3186280 to 7d9b489 Compare February 5, 2025 15:34
@Lyndon-Li
Copy link
Contributor

@kaovilai
There are some UT failures, please help to fix them.

Signed-off-by: Tiger Kaovilai <[email protected]>

comment update

Signed-off-by: Tiger Kaovilai <[email protected]>

comment

Signed-off-by: Tiger Kaovilai <[email protected]>
Signed-off-by: Tiger Kaovilai <[email protected]>
Signed-off-by: Tiger Kaovilai <[email protected]>
Signed-off-by: Tiger Kaovilai <[email protected]>
Signed-off-by: Tiger Kaovilai <[email protected]>
Signed-off-by: Tiger Kaovilai <[email protected]>
Signed-off-by: Tiger Kaovilai <[email protected]>
Signed-off-by: Tiger Kaovilai <[email protected]>
@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch from 6370bbb to 3fb8c72 Compare February 20, 2025 22:40
sseago
sseago previously approved these changes Feb 20, 2025
Lyndon-Li
Lyndon-Li previously approved these changes Feb 21, 2025
@Lyndon-Li
Copy link
Contributor

@kaovilai
I think we can add some UT for the new logics added in this PR.

@kaovilai kaovilai dismissed stale reviews from Lyndon-Li and sseago via 6f34a56 February 21, 2025 19:55
@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch from 6f34a56 to c36122e Compare February 21, 2025 19:57
@kaovilai kaovilai requested review from Lyndon-Li and sseago February 21, 2025 20:00
@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch 2 times, most recently from 62e0d8f to e01fcc4 Compare February 21, 2025 20:19
@kaovilai kaovilai force-pushed the configKopiaMaintInterval branch from e01fcc4 to 178b6e3 Compare February 21, 2025 20:22
@Lyndon-Li Lyndon-Li merged commit 564e774 into vmware-tanzu:main Feb 25, 2025
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Configurable Kopia Maintenance Interval
4 participants