Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pods are recreated because of Modification in CSO #948

Open
saurabhwani5 opened this issue Mar 9, 2023 · 3 comments
Open

Pods are recreated because of Modification in CSO #948

saurabhwani5 opened this issue Mar 9, 2023 · 3 comments
Labels
Customer Impact: Localized low impact (2) Temporary / limited perf impact, unnecessary failovers, issues occur while in degraded state Customer Probability: Low (1) Issue only occurs during failure condition - disk, server, network, test assert, ... Found In: 2.9.0 For Bug issues to identify what release level issue was found in 2.9.0 Severity: 3 Indicates the the issue is on the priority list for next milestone. Type: Bug Indicates issue is an undesired behavior, usually caused by code error.

Comments

@saurabhwani5
Copy link
Member

saurabhwani5 commented Mar 9, 2023 •
edited by Jainbrt
Loading

Describe the bug

After installing CSI 2.9.0(dev) while running the CSI Automation after sometime in few cases one error message is shown as

2023-03-07T14:49:38.626Z        ERROR   csiscaleoperator_controller.Reconcile   Reconciler Client.Update() failed       {"error": "Operation cannot be fulfilled on csiscaleoperators.csi.ibm.com \"ibm-spectrum-scale-csi\": the object has been modified; please apply your changes to the latest version and try again"}
2023-03-07T14:49:38.660Z        ERROR   controller.csiscaleoperator     Reconciler error        {"reconciler group": "csi.ibm.com", "reconciler kind": "CSIScaleOperator", "name": "ibm-spectrum-scale-csi", "namespace": "ibm-spectrum-scale-csi-driver", "error": "Operation cannot be fulfilled on csiscaleoperators.csi.ibm.com \"ibm-spectrum-scale-csi\": the object has been modified; please apply your changes to the latest version and try again"}

How to Reproduce?

  1. Install CSI 2.9.0
[root@power-csi-master tools]# oc get pods
NAME                                                  READY   STATUS    RESTARTS      AGE
ibm-spectrum-scale-csi-attacher-5885c459cb-dt2dq      1/1     Running   0             46h
ibm-spectrum-scale-csi-attacher-5885c459cb-mrxpk      1/1     Running   0             46h
ibm-spectrum-scale-csi-fw45h                          3/3     Running   0             46h
ibm-spectrum-scale-csi-operator-6dc7dd6655-x7pnf      1/1     Running   2 (46h ago)   4d21h
ibm-spectrum-scale-csi-provisioner-6b69d67cff-k55ls   1/1     Running   1 (46h ago)   46h
ibm-spectrum-scale-csi-resizer-6cc9f547d-j8mvc        1/1     Running   1 (36h ago)   46h
ibm-spectrum-scale-csi-snapshotter-6cf9b7d69b-zfgx5   1/1     Running   1 (46h ago)   46h
ibm-spectrum-scale-csi-zbd7v                          3/3     Running   0             46h

[root@power-csi-master tools]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.9.0     True
[root@power-csi-master tools]# oc describe pods | grep quay
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.9.0-020323
    Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:48716911136ff87cd1b107aa3bfcc09b09dfb744afa82ee6e2a84ac39b7913ee
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator:v2.9.0-020323
    Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:3c280b1640cac6d96d9ceefab035d7669ea91cc072f4a7debe8949714725f1f7
      CSI_DRIVER_IMAGE:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.9.0-020323
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver:v2.9.0-020323
    Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:48716911136ff87cd1b107aa3bfcc09b09dfb744afa82ee6e2a84ac39b7913ee
  1. While running full automation found that few test cases are failing (attaching in report) with that adding logs

Expected behavior

Pod Should not restart here
@saurabhwani5 saurabhwani5 added the Type: Bug Indicates issue is an undesired behavior, usually caused by code error. label Mar 9, 2023
@saurabhwani5
Copy link
Member Author

Uploaded logs in D.948

@saurabhwani5 saurabhwani5 changed the title Pods are restarted because of Modification in CSO Pods are recreated because of Modification in CSO Mar 13, 2023
@saurabhwani5 saurabhwani5 added Severity: 3 Indicates the the issue is on the priority list for next milestone. Customer Probability: Low (1) Issue only occurs during failure condition - disk, server, network, test assert, ... Customer Impact: Localized low impact (2) Temporary / limited perf impact, unnecessary failovers, issues occur while in degraded state Found In: 2.9.0 For Bug issues to identify what release level issue was found in 2.9.0 labels Mar 15, 2023
@deeghuge
Copy link
Member

deeghuge commented Feb 2, 2024

@saurabhwani5 please do keep watch on this one. If recreated easily then we can have a look.

@deeghuge
Copy link
Member

@saurabhwani5 please check if this is valid with latest code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Customer Impact: Localized low impact (2) Temporary / limited perf impact, unnecessary failovers, issues occur while in degraded state Customer Probability: Low (1) Issue only occurs during failure condition - disk, server, network, test assert, ... Found In: 2.9.0 For Bug issues to identify what release level issue was found in 2.9.0 Severity: 3 Indicates the the issue is on the priority list for next milestone. Type: Bug Indicates issue is an undesired behavior, usually caused by code error.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@deeghuge @saurabhwani5