Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OADP-3736 Need to set data-mover-prepare-timeout parameter in DPA #1368

Closed
libingwang123 opened this issue Mar 19, 2024 · 8 comments · May be fixed by #1657
Closed

OADP-3736 Need to set data-mover-prepare-timeout parameter in DPA #1368

libingwang123 opened this issue Mar 19, 2024 · 8 comments · May be fixed by #1657

Comments

@libingwang123
Copy link

We are using the OADP built-in dataMover to backup our ODF file system.
The backup worked well for small amount of file system data. When the data becomes bigger, the backup failed after 30 minutes.
The dataupload failed with status

status:
  message: timeout on preparing data upload
  phase: Failed
  progress: {}

We looked at https://velero.io/docs/main/csi-snapshot-data-movement/#customized-data-movers, and found

After the volume is created from the CSI snapshot, Velero built-in data mover waits for Kubernetes to provision the volume, this may take some time varying from storage providers, but if the provision cannot be finished in a given time, Velero built-in data mover will cancel this DataUpload CR. The timeout is configurable through a node-agent’s parameter data-mover-prepare-timeout, the default value is 30 minutes.

We are using DPA to backup. Here is the configuration of the DPA

  configuration:
    nodeAgent:
      enable: true
      uploaderType: restic
    velero:
      defaultPlugins:
        - openshift
        - csi
        - aws

We are not able to setup the data-mover-prepare-timeout from with the DPA. It seems that it is not available in DPA CRD.
@shubham-pampattiwar
Copy link
Member

@libingwang123 Thanks for reporting the issue. Currently we do not have an option to set the data-mover-prepare-timeout via DPA CR. This is probably the first time we are seeing a user needing to customize this timeout, we will definitely add the task of enabling data-mover-prepare-timeout field via DPA in our Operator Development roadmap.

@libingwang123
Copy link
Author

libingwang123 commented Mar 20, 2024 •
edited
Loading

@shubham-pampattiwar Thanks for the quick response.
Is there an estimated deliver date for this feature in the new OADP version?
In the meanwhile, is there any way to work this around? E.g., once datauploader is up, is there any way to edit the nodeagent injecting this parameter? Thanks.

@sikhlaqu
Copy link

Is there any update on this issue.
Any ETA when this will be available part of the new OADP Release ?

Note: This FLAG is already available at Velero side, but there is no way to pass it at OADP end. Hence the OADP backup timesout after 30min. Today its kind of blocking for the OADP Users.

@kaovilai
Copy link
Member

kaovilai commented May 13, 2024 •
edited
Loading

Per https://issues.redhat.com/browse/OADP-3736 this work is aligned to OADP 1.4 release.

@kaovilai kaovilai changed the title Need to set data-mover-prepare-timeout parameter in DPA OADP-3736 Need to set data-mover-prepare-timeout parameter in DPA May 13, 2024
@kaovilai kaovilai added this to OADP May 13, 2024
@libingwang123
Copy link
Author

Thanks @kaovilai. Is there a GA date on OADP 1.4?

@keltakmaster
Copy link

Hi! We need this update too! Thanks

@kaovilai
Copy link
Member

dupe #1411 and fixed by #1450

@kaovilai
Copy link
Member

1.3 cherrypick #1457

mpryc added a commit to mpryc/oadp-operator that referenced this issue Mar 6, 2025
@mpryc
Add the dataMoverPrepareTimeout and resourceTimeout to the DPA's Node…
39905c5 
…-Agent

Fixes openshift#1368

Adds the following fields to the Node-Agent within the DPA:
 - dataMoverPrepareTimeout
   How long to wait for preparing a DataUpload/DataDownload. Default is 30 minutes.
 - resourceTimeout
   How long to wait for resource processes which are not covered by other specific
   timeout parameters. Default is 10 minutes.

Signed-off-by: Michal Pryc <[email protected]>
@mpryc mpryc mentioned this issue Mar 6, 2025
Open
mpryc added a commit to mpryc/oadp-operator that referenced this issue Mar 6, 2025
@mpryc
Add the dataMoverPrepareTimeout and resourceTimeout to the DPA's Node…
8d45879 
…-Agent

Fixes openshift#1368

Adds the following fields to the Node-Agent within the DPA:
 - dataMoverPrepareTimeout
   How long to wait for preparing a DataUpload/DataDownload. Default is 30 minutes.
 - resourceTimeout
   How long to wait for resource processes which are not covered by other specific
   timeout parameters. Default is 10 minutes.

Signed-off-by: Michal Pryc <[email protected]>
mpryc added a commit to mpryc/oadp-operator that referenced this issue Mar 6, 2025
@mpryc
Add the dataMoverPrepareTimeout and resourceTimeout to the DPA's Node…
7c6ccf7 
…-Agent

Fixes openshift#1368

Adds the following fields to the Node-Agent within the DPA:
 - dataMoverPrepareTimeout
   How long to wait for preparing a DataUpload/DataDownload. Default is 30 minutes.
 - resourceTimeout
   How long to wait for resource processes which are not covered by other specific
   timeout parameters. Default is 10 minutes.

Signed-off-by: Michal Pryc <[email protected]>
mpryc added a commit to mpryc/oadp-operator that referenced this issue Mar 6, 2025
@mpryc
Add the dataMoverPrepareTimeout and resourceTimeout to the DPA's Node…
e9e506f 
…-Agent

Fixes openshift#1368

Adds the following fields to the Node-Agent within the DPA:
 - dataMoverPrepareTimeout
   How long to wait for preparing a DataUpload/DataDownload. Default is 30 minutes.
 - resourceTimeout
   How long to wait for resource processes which are not covered by other specific
   timeout parameters. Default is 10 minutes.

Signed-off-by: Michal Pryc <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add the dataMoverPrepareTimeout and resourceTimeout to the Node-Agent mpryc/oadp-operator
5 participants
@sikhlaqu @kaovilai @keltakmaster @shubham-pampattiwar @libingwang123