arm64 robustness test failure

[serathius]

Which Github Action / Prow Jobs are flaking?

Robustness nightly on arm

Which tests are flaking?

TestRobustnessExploratory/Etcd/HighTraffic/ClusterOfSize3 x2
TestRobustnessExploratory/Kubernetes/LowTraffic/ClusterOfSize3
TestRobustnessExploratory/Etcd/LowTraffic/ClusterOfSize3

• Timeouts

Github Action / Prow Job link

https://github.com/etcd-io/etcd/actions/runs/8153962092/job/22286380964

Reason for failure (if possible)

7/18 flake ratio, all on arm64.

In most cases logs imply too low qps, which might be sign of lower performance, or issue with recovering from failure injection.

I have seen previously failure cases with logs reporting broken snapshots sent between members. So there might be an underlying correctness issue on arm64.

Anything else we need to know?

If arm64 tests are not stabilized, I will consider disabling it due to lack of support.

[ah8ad3]

If it's ok i can take a look at them. Maybe i can find the issue.

[jmhbnz]

Hey @ah8ad3 - Thanks for your interest, I will take this one. I believe it's a performance related challenge with our arm64 CI infrastructure.

Please feel free to look into any failures on arm64 that don't relate to qps to low, these will be more interesting flakes to explore.

/assign

[ah8ad3]

Thanks for the info @jmhbnz . OK i'll do that. GL on this one.

[jmhbnz]

Closing - The performance issues we historically observed for arm64 specifically in robustness tests should now be alleviated through our migration to prow for these workflows.

Reviewing recent results:

• ci-etcd-robustness-release34-arm64 https://testgrid.k8s.io/sig-etcd-robustness#ci-etcd-robustness-release34-arm64 5/5
  

• ci-etcd-robustness-release35-arm64 https://testgrid.k8s.io/sig-etcd-robustness#ci-etcd-robustness-release35-arm64 5/5
  

• ci-etcd-robustness-main-arm64 https://testgrid.k8s.io/sig-etcd-robustness#ci-etcd-robustness-main-arm64 3/5
  Note failures here are Linearization has timed out {"duration": "5m0.0063106s"} which we also see for amd64 so in the same prow cluster so I don't believe that to be an arm64 specific issue.

• pull-etcd-robustness-arm64 https://testgrid.k8s.io/sig-etcd-robustness#pull-etcd-robustness-arm64 8/10
  

• ci-etcd-robustness-arm64 https://testgrid.k8s.io/sig-etcd-robustness#ci-etcd-robustness-arm64 0/10
  This one is definitely still an issue, however investigation/work to address it is being tracked under Migrate the robustness tests to prow #18136 so I think this old issue should still be closed, or if we want to track it separately from the migration I think we should open a fresh issue that does not relate to GitHub actions.