ovn-ic-server recover wait longer time

[changluyi]

Pull Request

• Make sure you have followed Kube-OVN Code Style.

What type of this PR

Examples of user facing changes:

出错的流水线
https://github.com/zhangzujian/kube-ovn/actions/runs/7969190992/job/21754799911

06:26:04 触发修改TS_NUM的环境变量

06:26:40 deployment才触发重启，环境变量生效。但是结果判定在06:26:39 就发生了。

• Features
• Bug fixes
• Docs
• Tests

Which issue(s) this PR fixes

Fixes #(issue-number)

[zhangzujian]

Another failure:

• [FAILED] [22.510 seconds]
[CNI:Kube-OVN] [group:ovn-ic] [It] should be able to update az name [Conformance] [Conformance]
/home/runner/work/kube-ovn/kube-ovn/test/e2e/framework/framework.go:199

  Timeline >>
  STEP: Setting kubernetes context @ 02/20/24 10:21:34.257
  Feb 20 10:21:34.258: INFO: Switching context to kind-kube-ovn1
  STEP: Creating a kubernetes client @ 02/20/24 10:21:34.262
  Feb 20 10:21:34.262: INFO: >>> kubeConfig: /home/runner/.kube/config
  STEP: Building a namespace api object, basename ovn-ic @ 02/20/24 10:21:34.263
  STEP: Waiting for a default service account to be provisioned in namespace @ 02/20/24 10:21:34.285
  STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 02/20/24 10:21:34.29
  STEP: Setting kubernetes context @ 02/20/24 10:21:34.293
  Feb 20 10:21:34.294: INFO: Switching context to kind-kube-ovn
  STEP: Creating a kubernetes client @ 02/20/24 10:21:34.299
  Feb 20 10:21:34.299: INFO: >>> kubeConfig: /home/runner/.kube/config
  STEP: Building a namespace api object, basename ovn-ic @ 02/20/24 10:21:34.3
  STEP: Waiting for a default service account to be provisioned in namespace @ 02/20/24 10:21:34.31
  STEP: Waiting for kube-root-ca.crt to be provisioned in namespace @ 02/20/24 10:21:34.313
  STEP: fetching the ConfigMap in cluster kube-ovn1 @ 02/20/24 10:21:34.315
  STEP: fetching the ConfigMap in cluster kube-ovn @ 02/20/24 10:21:34.317
  STEP: patching the ConfigMap in cluster kube-ovn1 @ 02/20/24 10:21:34.319
  STEP: Waiting for new az names to be applied @ 02/20/24 10:21:34.322
  STEP: Switching context to kind-kube-ovn1 @ 02/20/24 10:21:44.328
  Feb 20 10:21:44.328: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl config use-context kind-kube-ovn1'
  Feb 20 10:21:44.380: INFO: stderr: ""
  Feb 20 10:21:44.381: INFO: stdout: "Switched to context \"kind-kube-ovn1\".\n"
  STEP: Executing "ovn-appctl -t ovn-controller inc-engine/recompute" in pod kube-system/ovs-ovn-8t2jk @ 02/20/24 10:21:44.381
  Feb 20 10:21:44.381: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl --namespace=kube-system exec ovs-ovn-8t2jk -- /bin/sh -x -c ovn-appctl -t ovn-controller inc-engine/recompute'
  Feb 20 10:21:44.509: INFO: stderr: "+ ovn-appctl -t ovn-controller inc-engine/recompute\n"
  Feb 20 10:21:44.509: INFO: stdout: ""
  Feb 20 10:21:44.509: INFO: stdout: 
  STEP: Switching context to kind-kube-ovn1 @ 02/20/24 10:21:44.509
  Feb 20 10:21:44.509: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl config use-context kind-kube-ovn1'
  Feb 20 10:21:44.557: INFO: stderr: ""
  Feb 20 10:21:44.557: INFO: stdout: "Switched to context \"kind-kube-ovn1\".\n"
  STEP: Executing "ovn-appctl -t ovn-controller inc-engine/recompute" in pod kube-system/ovs-ovn-br97m @ 02/20/24 10:21:44.557
  Feb 20 10:21:44.557: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl --namespace=kube-system exec ovs-ovn-br97m -- /bin/sh -x -c ovn-appctl -t ovn-controller inc-engine/recompute'
  Feb 20 10:21:44.721: INFO: stderr: "+ ovn-appctl -t ovn-controller inc-engine/recompute\n"
  Feb 20 10:21:44.722: INFO: stdout: ""
  Feb 20 10:21:44.722: INFO: stdout: 
  STEP: Switching context to kind-kube-ovn1 @ 02/20/24 10:21:44.722
  Feb 20 10:21:44.722: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl config use-context kind-kube-ovn1'
  Feb 20 10:21:44.763: INFO: stderr: ""
  Feb 20 10:21:44.763: INFO: stdout: "Switched to context \"kind-kube-ovn1\".\n"
  STEP: Executing "ovn-appctl -t ovn-controller inc-engine/recompute" in pod kube-system/ovs-ovn-hc9km @ 02/20/24 10:21:44.763
  Feb 20 10:21:44.763: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl --namespace=kube-system exec ovs-ovn-hc9km -- /bin/sh -x -c ovn-appctl -t ovn-controller inc-engine/recompute'
  Feb 20 10:21:44.881: INFO: stderr: "+ ovn-appctl -t ovn-controller inc-engine/recompute\n"
  Feb 20 10:21:44.881: INFO: stdout: ""
  Feb 20 10:21:44.881: INFO: stdout: 
  STEP: Switching context to kind-kube-ovn1 @ 02/20/24 10:21:44.882
  Feb 20 10:21:44.882: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl config use-context kind-kube-ovn1'
  Feb 20 10:21:44.925: INFO: stderr: ""
  Feb 20 10:21:44.925: INFO: stdout: "Switched to context \"kind-kube-ovn1\".\n"
  STEP: Executing "ovn-appctl -t ovn-controller inc-engine/recompute" in pod kube-system/ovs-ovn-zktfd @ 02/20/24 10:21:44.925
  Feb 20 10:21:44.925: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl --namespace=kube-system exec ovs-ovn-zktfd -- /bin/sh -x -c ovn-appctl -t ovn-controller inc-engine/recompute'
  Feb 20 10:21:45.120: INFO: stderr: "+ ovn-appctl -t ovn-controller inc-engine/recompute\n"
  Feb 20 10:21:45.120: INFO: stdout: ""
  Feb 20 10:21:45.120: INFO: stdout: 
  STEP: Ensuring logical switch ts exists in cluster kube-ovn1 @ 02/20/24 10:21:47.121
  STEP: Switching context to kind-kube-ovn1 @ 02/20/24 10:21:47.121
  Feb 20 10:21:47.121: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl config use-context kind-kube-ovn1'
  Feb 20 10:21:47.163: INFO: stderr: ""
  Feb 20 10:21:47.163: INFO: stdout: "Switched to context \"kind-kube-ovn1\".\n"
  STEP: Executing "kubectl ko nbctl show ts" @ 02/20/24 10:21:47.163
  Feb 20 10:21:47.163: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl ko nbctl show ts'
  Feb 20 10:21:47.580: INFO: stderr: ""
  Feb 20 10:21:47.581: INFO: stdout: "switch 24337163-9782-4821-89cb-a85e2fb2409b (ts)\n    port ts-az0\n        type: remote\n        addresses: [\"00:00:00:19:18:63 169.254.100.94/24\"]\n    port ts-az9453\n        type: router\n        router-port: az9453-ts\n"
  STEP: Creating pod pod-116308749 in cluster kube-ovn1 @ 02/20/24 10:21:47.581
  W0220 10:21:47.585501   87844 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "container" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "container" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "container" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
  STEP: Creating pod pod-134085313 in cluster kube-ovn @ 02/20/24 10:21:49.592
  W0220 10:21:49.596590   87844 warnings.go:70] would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "container" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "container" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "container" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "container" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
  STEP: Checking connection from cluster kube-ovn1 to cluster kube-ovn via ipv4 @ 02/20/24 10:21:51.602
  STEP: Switching context to kind-kube-ovn1 @ 02/20/24 10:21:51.602
  Feb 20 10:21:51.602: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl config use-context kind-kube-ovn1'
  Feb 20 10:21:51.644: INFO: stderr: ""
  Feb 20 10:21:51.644: INFO: stdout: "Switched to context \"kind-kube-ovn1\".\n"
  STEP: Executing "curl -q -s --connect-timeout 5 10.16.0.23:8343/clientip" in pod ovn-ic-[529](https://github.com/zhangzujian/kube-ovn/actions/runs/7971375908/job/21762176315#step:14:530)1/pod-116308749 @ 02/20/24 10:21:51.644
  Feb 20 10:21:51.644: INFO: Running '/opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl --namespace=ovn-ic-5291 exec pod-116308749 -- /bin/sh -x -c curl -q -s --connect-timeout 5 10.16.0.23:8343/clientip'
  Feb 20 10:21:56.760: INFO: rc: 28
  Feb 20 10:21:56.760: INFO: stdout: 
  Feb 20 10:21:56.760: INFO: Unexpected error: 
      <exec.CodeExitError>: 
      error running /opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl --namespace=ovn-ic-5291 exec pod-116308749 -- /bin/sh -x -c curl -q -s --connect-timeout 5 10.16.0.23:8343/clientip:
      Command stdout:
      
      stderr:
      + curl -q -s --connect-timeout 5 10.16.0.23:8343/clientip
      command terminated with exit code 28
      
      error:
      exit status 28
      {
          Err: <*errors.errorString | 0xc00094a020>{
              s: "error running /opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl --namespace=ovn-ic-5291 exec pod-116308749 -- /bin/sh -x -c curl -q -s --connect-timeout 5 10.16.0.23:8343/clientip:\nCommand stdout:\n\nstderr:\n+ curl -q -s --connect-timeout 5 10.16.0.23:8343/clientip\ncommand terminated with exit code 28\n\nerror:\nexit status 28",
          },
          Code: 28,
      }
  [FAILED] in [It] - /home/runner/go/pkg/mod/k8s.io/[email protected]/test/e2e/framework/pod/output/output.go:95 @ 02/20/24 10:21:56.76
  STEP: Destroying namespace "ovn-ic-8301" for this suite. @ 02/20/24 10:21:56.761
  STEP: Destroying namespace "ovn-ic-5291" for this suite. @ 02/20/24 10:21:56.764
  << Timeline

  [FAILED] error running /opt/hostedtoolcache/kind/v0.22.0/amd64/kubectl/bin/kubectl --namespace=ovn-ic-5291 exec pod-116308749 -- /bin/sh -x -c curl -q -s --connect-timeout 5 10.16.0.23:8343/clientip:
  Command stdout:

  stderr:
  + curl -q -s --connect-timeout 5 10.16.0.23:8343/clientip
  command terminated with exit code 28

  error:
  exit status 28
  In [It] at: /home/runner/go/pkg/mod/k8s.io/[email protected]/test/e2e/framework/pod/output/output.go:95 @ 02/20/24 10:21:56.76