[Bug]: delta logs - Object name contains unsupported characters.

[gmoshiko]

Current Behavior

right now after trying to restore, the collection with insert logs seems to be ok, but when reach to delta logs we can see it tries to copy to the data to bad location.
this doesnt happen when using on collections without delta logs...

logs:

ck="github.com/zilliztech/milvus-backup/core/storage.(*MinioChunkManager).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/minio_chunk_manager.g
o:447\ngithub.com/zilliztech/milvus-backup/core/storage.(*Copier).copyRemote\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:261\ngithub.c
om/zilliztech/milvus-backup/core/storage.(*Copier).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:242\ngithub.com/zilliztech/milvus-
backup/core.(*BackupContext).executeRestoreCollectionTask.func3.1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:610\ngithub.
com/zilliztech/milvus-backup/internal/util/retry.Do\n\t/home/runner/work/milvus-backup/milvus-backup/internal/util/retry/retry.go:37\ngithub.com/zilliztech/milvus
-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:609\ngithub.c
om/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func9\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_bac
kup.go:754\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpo
ol.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2024/11/21 13:14:35.021 +00:00] [DEBUG] [storage/copier.go:260] [copyRemote] [srcBucket=backup-nocrooss] [destBucket=milvus-bucket-data] [key=temp_data/llm_prod_
1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347/452408614719061504] [destKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm
_prod_1_chunks_restore/llm_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504]
[2024/11/21 13:14:35.052 +00:00] [ERROR] [storage/minio_chunk_manager.go:447] ["copyObject error"] [srcObjectKey=temp_data/llm_prod_1_chunks_backup/binlogs/delta_
log/452408614878819909/-1/452408614719061347/452408614719061504] [dstObjectKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm_prod_1_chunks_restore/ll
m_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504] [error="Object name contains unsupported characters."] [sta
ck="github.com/zilliztech/milvus-backup/core/storage.(*MinioChunkManager).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/minio_chunk_manager.g
o:447\ngithub.com/zilliztech/milvus-backup/core/storage.(*Copier).copyRemote\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:261\ngithub.c
om/zilliztech/milvus-backup/core/storage.(*Copier).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:242\ngithub.com/zilliztech/milvus-
backup/core.(*BackupContext).executeRestoreCollectionTask.func3.1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:610\ngithub.
com/zilliztech/milvus-backup/internal/util/retry.Do\n\t/home/runner/work/milvus-backup/milvus-backup/internal/util/retry/retry.go:37\ngithub.com/zilliztech/milvus
-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:609\ngithub.c
om/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func9\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_bac
kup.go:754\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpo
ol.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2024/11/21 13:14:39.068 +00:00] [DEBUG] [storage/copier.go:260] [copyRemote] [srcBucket=backup-nocrooss] [destBucket=milvus-bucket-data] [key=temp_data/llm_prod_
1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347/452408614719061504] [destKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm
_prod_1_chunks_restore/llm_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504]
[2024/11/21 13:14:39.106 +00:00] [ERROR] [storage/minio_chunk_manager.go:447] ["copyObject error"] [srcObjectKey=temp_data/llm_prod_1_chunks_backup/binlogs/delta_
log/452408614878819909/-1/452408614719061347/452408614719061504] [dstObjectKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm_prod_1_chunks_restore/ll
m_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504] [error="Object name contains unsupported characters."] [sta
ck="github.com/zilliztech/milvus-backup/core/storage.(*MinioChunkManager).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/minio_chunk_manager.g
o:447\ngithub.com/zilliztech/milvus-backup/core/storage.(*Copier).copyRemote\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:261\ngithub.c
[2024/11/21 13:14:28.808 +00:00] [INFO] [common/workerpool.go:70] ["restore l0 segment "] [backup_db_name=llm] [backup_collection_name=llm] [target_db_name=llm] [
target_collection_name=llm_prod_1_chunks_restore] [skipDiskQuotaCheck=false] [maxShardNum=0] [files=temp_data/llm_prod_1_chunks_backup/binlogs/delta_log/452408614
878819909/-1/452408614719061347]
[2024/11/21 13:14:28.808 +00:00] [INFO] [core/backup_impl_restore_backup.go:754] ["milvus and backup store in different bucket, copy the data first"] [backup_db_n
ame=llm] [backup_collection_name=llm] [target_db_name=llm] [target_collection_name=llm_prod_1_chunks_restore] [skipDiskQuotaCheck=false] [maxShardNum=0] [files="[
temp_data/llm_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347]"] [copyDataPath=restore-temp-restore_2024_11_21_13_09_51_495718634-
llm-llm_prod_1_chunks_restore/]
[2024/11/21 13:14:28.808 +00:00] [DEBUG] [core/backup_impl_restore_backup.go:754] ["Copy temporary restore file"] [backup_db_name=llm] [backup_collection_name=llm
] [target_db_name=llm] [target_collection_name=llm_prod_1_chunks_restore] [skipDiskQuotaCheck=false] [maxShardNum=0] [from=temp_data/llm_prod_1_chunks_backup/binl
ogs/delta_log/452408614878819909/-1/452408614719061347] [to=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm_prod_1_chunks_restore/llm_prod_1_chunks_bac
kup/binlogs/delta_log/452408614878819909/-1/452408614719061347/]
[2024/11/21 13:14:28.877 +00:00] [DEBUG] [storage/copier.go:260] [copyRemote] [srcBucket=backup-nocrooss] [destBucket=milvus-bucket-data] [key=temp_data/llm_prod_
1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347/452408614719061504] [destKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm
_prod_1_chunks_restore/llm_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504]
[2024/11/21 13:14:28.957 +00:00] [ERROR] [storage/minio_chunk_manager.go:447] ["copyObject error"] [srcObjectKey=temp_data/llm_prod_1_chunks_backup/binlogs/delta_
log/452408614878819909/-1/452408614719061347/452408614719061504] [dstObjectKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm_prod_1_chunks_restore/ll
m_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504] [error="Object name contains unsupported characters."] [sta
ck="github.com/zilliztech/milvus-backup/core/storage.(*MinioChunkManager).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/minio_chunk_manager.g
o:447\ngithub.com/zilliztech/milvus-backup/core/storage.(*Copier).copyRemote\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:261\ngithub.c
om/zilliztech/milvus-backup/core/storage.(*Copier).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:242\ngithub.com/zilliztech/milvus-
backup/core.(*BackupContext).executeRestoreCollectionTask.func3.1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:610\ngithub.
com/zilliztech/milvus-backup/internal/util/retry.Do\n\t/home/runner/work/milvus-backup/milvus-backup/internal/util/retry/retry.go:37\ngithub.com/zilliztech/milvus
-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:609\ngithub.c
om/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func9\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_bac
kup.go:754\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpo
ol.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2024/11/21 13:14:30.973 +00:00] [DEBUG] [storage/copier.go:260] [copyRemote] [srcBucket=backup-nocrooss] [destBucket=milvus-bucket-data] [key=temp_data/llm_prod_
1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347/452408614719061504] [destKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm
_prod_1_chunks_restore/llm_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504]
[2024/11/21 13:14:31.004 +00:00] [ERROR] [storage/minio_chunk_manager.go:447] ["copyObject error"] [srcObjectKey=temp_data/llm_prod_1_chunks_backup/binlogs/delta_
log/452408614878819909/-1/452408614719061347/452408614719061504] [dstObjectKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm_prod_1_chunks_restore/ll
m_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504] [error="Object name contains unsupported characters."] [staom/zilliztech/milvus-backup/core/storage.(*Copier).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:242\ngithub.com/zilliztech/milvus-
backup/core.(*BackupContext).executeRestoreCollectionTask.func3.1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:610\ngithub.
com/zilliztech/milvus-backup/internal/util/retry.Do\n\t/home/runner/work/milvus-backup/milvus-backup/internal/util/retry/retry.go:37\ngithub.com/zilliztech/milvus
-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:609\ngithub.c
om/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func9\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_bac
kup.go:754\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpo
ol.go:70\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]
[2024/11/21 13:14:43.126 +00:00] [DEBUG] [storage/copier.go:260] [copyRemote] [srcBucket=backup-nocrooss] [destBucket=milvus-bucket-data] [key=temp_data/llm_prod_
1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347/452408614719061504] [destKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm
_prod_1_chunks_restore/llm_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504]
[2024/11/21 13:14:43.161 +00:00] [ERROR] [storage/minio_chunk_manager.go:447] ["copyObject error"] [srcObjectKey=temp_data/llm_prod_1_chunks_backup/binlogs/delta_
log/452408614878819909/-1/452408614719061347/452408614719061504] [dstObjectKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm_prod_1_chunks_restore/ll
m_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504] [error="Object name contains unsupported characters."] [sta
ck="github.com/zilliztech/milvus-backup/core/storage.(*MinioChunkManager).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/minio_chunk_manager.g
o:447\ngithub.com/zilliztech/milvus-backup/core/storage.(*Copier).copyRemote\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:261\ngithub.c
om/zilliztech/milvus-backup/core/storage.(*Copier).Copy\n\t/home/runner/work/milvus-backup/milvus-backup/core/storage/copier.go:242\ngithub.com/zilliztech/milvus-
backup/core.(*BackupContext).executeRestoreCollectionTask.func3.1\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:610\ngithub.
com/zilliztech/milvus-backup/internal/util/retry.Do\n\t/home/runner/work/milvus-backup/milvus-backup/internal/util/retry/retry.go:37\ngithub.com/zilliztech/milvus
-backup/core.(*BackupContext).executeRestoreCollectionTask.func3\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_backup.go:609\ngithub.c
om/zilliztech/milvus-backup/core.(*BackupContext).executeRestoreCollectionTask.func9\n\t/home/runner/work/milvus-backup/milvus-backup/core/backup_impl_restore_bac
kup.go:754\ngithub.com/zilliztech/milvus-backup/internal/common.(*WorkerPool).work.func1\n\t/home/runner/work/milvus-backup/milvus-backup/internal/common/workerpo
ol.go:70\ngolang.org/x/sync/errgroup.(*Gro).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75"]

you can see here the double slash (//) on the "destKey" which should be only once (/) and when trying to copy data from minio like that you get "Object name contains unsupported characters."

[2024/11/21 13:14:43.126 +00:00] [DEBUG] [storage/copier.go:260] [copyRemote] [srcBucket=backup-nocrooss] [destBucket=milvus-bucket-data] [key=temp_data/llm_prod_
1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347/452408614719061504] [destKey=restore-temp-restore_2024_11_21_13_09_51_495718634-llm-llm
_prod_1_chunks_restore/llm_prod_1_chunks_backup/binlogs/delta_log/452408614878819909/-1/452408614719061347//452408614719061504]

Expected Behavior

to work also on collections that had delete operation before.

Steps To Reproduce

1. backup collection with delta logs
2. try to restore
3. not sure if related to cross storage feature

Environment

Milvus(cluster): 2.4.4
Milvus-backup: v0.4.24

Anything else?

No response

[wayblink]

@gmoshiko Thanks for report this issue, we will optimize the path concat compatibility.

[gmoshiko]

@wayblink Hey, any update on timeline for this ? Thanks !

[11223300]

@gmoshiko Hello, after this situation occurs, how can the collection be temporarily restored?

[balazik]

We have encountered the same error while running the restore operation using the Milvus-backup tool. Can't restore a 500M vectors DB, because there is a double slash (//) in delta_logs path.

Below is the error message:

[error="All attempts results:
attempt #1:storage: minio upload object restore-temp-restore_2025_01_23_15_49_17_128868281-default-platform_semantic_search/platform_semantic_search_20250123/binlogs/delta_log/452555602791241734/452555602791241835/453016256345383812//453016256345383812/453016256345385570 Object name contains unsupported characters.
attempt #2:storage: minio upload object

Steps to Reproduce:

1. Run the restore operation using the Milvus-backup tool.
2. Observe the error message indicating unsupported characters in the object name.

Environment:

• Milvus-backup tool version: v0.4.30
• Operating System: Ubuntu 22.04.2 LTS