How to migrate data from docker compose volumes files to k8s ?

[AnyangAngus]

I have deployed docker compose milvus for 1 year. But this week My docker compose milvus server meet a bug, all the collections can not be loaded and face a Error: 4 DEADLINE_EXCEEDED: Deadline exceeded. I deploy the server to k8s.
Now how can I migrate data from docker compose volumes on file system to k8s milvus server?

[yhmo]

If a collection cannot be loaded, there must be a critical error in the log. It is better to find out what the error is.
milvus-backup tool is a recommended tool to backup/restore data between milvus instances.
https://milvus.io/docs/milvus_backup_overview.md

[yhmo]

milvus-backup needs to connect both minio and milvus server.

backup: connect the source minio and the source milvus server
restore: connect the target minio and the target milvus server

[AnyangAngus]

milvus-backup needs to connect both minio and milvus server.

backup: connect the source minio and the source milvus server restore: connect the target minio and the target milvus server

@yhmo If my backup milvus server can not load connections , Can the milvus-backup migrate data will ?

[yhmo]

According to the source code of milvus-backup, it doesn't require the collection to be loaded.

[AnyangAngus]

According to the source code of milvus-backup, it doesn't require the collection to be loaded.

@yhmo
My milvus standalone version is 2.3.0, the minio conf in docker-compose is :

  minio:
    container_name: milvus-minio
    image: minio/minio:RELEASE.2023-03-20T20-16-18Z
    environment:
      MINIO_ACCESS_KEY: minioadmin
      MINIO_SECRET_KEY: minioadmin
    ports:
      - "9001:9001"
      - "9000:9000"
    volumes:
      - ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
    command: minio server /minio_data --console-address ":9001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 2

when I try milvus-backup, the conf is

# Configures the system log output.
log:
  level: info # Only supports debug, info, warn, error, panic, or fatal. Default 'info'.
  console: true # whether print log to console
  file:
    rootPath: "logs/backup.log"

http:
  simpleResponse: true

# milvus proxy address, compatible to milvus.yaml
milvus:
  address: localhost
  port: 19530
  authorizationEnabled: false
  # tls mode values [0, 1, 2]
  # 0 is close, 1 is one-way authentication, 2 is two-way authentication.
  tlsMode: 0
  user: "root"
  password: "Milvus"

# Related configuration of minio, which is responsible for data persistence for Milvus.
minio:
  # Milvus storage configs, make them the same with milvus config
  storageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
  address: localhost # Address of MinIO/S3
  port: 9000   # Port of MinIO/S3
  accessKeyID: minioadmin  # accessKeyID of MinIO/S3
  secretAccessKey: minioadmin # MinIO/S3 encryption string
  useSSL: false # Access to MinIO/S3 with SSL
  useIAM: false
  iamEndpoint: ""
  bucketName: "a-bucket" # Milvus Bucket name in MinIO/S3, make it the same as your milvus instance
  rootPath: "files" # Milvus storage root path in MinIO/S3, make it the same as your milvus instance

  # Backup storage configs, the storage you want to put the backup data
  backupStorageType: "minio" # support storage type: local, minio, s3, aws, gcp, ali(aliyun), azure, tc(tencent)
  backupAddress: localhost # Address of MinIO/S3
  backupPort: 9000   # Port of MinIO/S3
  backupAccessKeyID: minioadmin  # accessKeyID of MinIO/S3
  backupSecretAccessKey: minioadmin # MinIO/S3 encryption string
  backupBucketName: "a-bucket" # Bucket name to store backup data. Backup data will store to backupBucketName/backupRootPath
  backupRootPath: "backup" # Rootpath to store backup data. Backup data will store to backupBucketName/backupRootPath

sh: ./milvus-backup create -c coll -n my_backup
I got the error msg:

 [ERROR] [core/backup_impl_create_backup.go:220] ["request backup collection does not exist: default.coll"]

but I can connect minio by minio client:

 mc config host add minio http://localhost:9000 minioadmin minioadmin
Added `minio` successfully.

[yhmo]

"request backup collection does not exist: default.coll" ---------------------- the collection named "coll" doesn't exist in the database named "default".

I guess the "coll" is in another database. Specify the database name in the command:
./milvus-backup create -d my_db_name -c coll -n my_backup