feat: data plane & envoyproxy resilience test suite

[alexwo]

What type of PR is this?
This pr adds a resilience test suite to cover the expected behavior of the data plane for both existing and new proxy instances during specific edge conditions.

Why do we use Cilium?
The default network plugin in Kubernetes-in-Docker (Kind) does not support enforcing network policies that restrict connectivity to local services, such as the Kubernetes API server. However, by disabling the default CNI and deploying Cilium as the CNI plugin, we can implement network policies that effectively control access to the API server. This approach allows for precise management of network traffic within the cluster, including the ability to block connectivity to the Kubernetes API server as needed.

New test suite
Having a new resilience test suite can be helpful due to the unique network setup and tests involved in resilience testing, which differ significantly from those of existing test suites.

For instance, we need a custom CNI only for this test case, other tests can still use the default.

Fault Injection: Resilience testing involves deliberately introducing failures to assess the system’s response and recovery capabilities. This approach is not typically encompassed by standard functional or performance test suites.

By maintaining a separate resilience test suite, we can effectively simulate and manage system stressors, and separate this concern from the other test cases we have.

Test suite is expected to pass once #4767 is merged.

Issue:
#4861

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 66.28%. Comparing base (d33b617) to head (50ca751).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #4862   +/-   ##
=======================================
  Coverage   66.28%   66.28%           
=======================================
  Files         209      209           
  Lines       31942    31942           
=======================================
+ Hits        21172    21174    +2     
+ Misses       9517     9516    -1     
+ Partials     1253     1252    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[alexwo]

/retest

[alexwo]

/retest

[zirain]

can you share more details about why we need cilium?

[alexwo]

The default network plugin in Kubernetes-in-Docker (Kind) does not support enforcing network policies that restrict connectivity to local services, such as the Kubernetes API server. However, by disabling the default CNI and deploying Cilium as the CNI plugin, we can implement network policies that effectively control access to the API server. This approach allows for precise management of network traffic within the cluster, including the ability to block connectivity to the Kubernetes API server as needed.

[zirain]

I recall kind support network policy as kubernetes-sigs/kind#842 closed.

[alexwo]

@zirain That’s correct, network policies are supported now. However, even the new local kind network adapter is currently not enforcing network policies that restrict connectivity to local services like the API server. As a result, the default kind adapter isn’t suitable for this specific use case.

[zirain]

will https://github.com/kubernetes-sigs/kube-network-policies help?

[alexwo]

I will give it a try, maybe it can do the work 👍

[alexwo]

@zirain I tested with kube-network-policies, but it was unable to block connectivity to the API server. In fact, I tried all the other available options as well. Currently, it seems this functionality is only achievable with the Cilium CNI.

[zirain]

why we need a new test suite instead of reuse existing one?

[alexwo]

Contributor

why we need a new test suite instead of reuse existing one?

Having a dedicated resilience test suite can be helpful due to the unique network setup and tests involved in resilience testing, which differ significantly from those of existing test suites.

For instance, we need a custom CNI only for this test case, other tests can still use the default.

Fault Injection: Resilience testing involves deliberately introducing failures to assess the system’s response and recovery capabilities. This approach is not typically encompassed by standard functional or performance test suites.

By maintaining a separate resilience test suite, we can effectively simulate and manage system stressors, and separate this concern from the other test cases we have.

Running this suite in parallel will keep us at good execution time.

[alexwo]

/retest

[zhaohuabing]

Overall, it looks great! This test will definitely improve the stability of EG!

I left a few minor comments.
It also seems the three Envoy Gateway tests can be merged into one.

[zhaohuabing]

I would prefer to make this as part of e2e if there's no special reason. Especailly don't want to introudce a whole new test suite framework.

@envoyproxy/gateway-maintainers please share your options.

It could be included in the "normal" test suite. I don't have a strong opinion on this.

[alexwo]

Overall, it looks great! This test will definitely improve the stability of EG!

I left a few minor comments. It also seems the three Envoy Gateway tests can be merged into one.

Thank you for your feedback. I’ve added more context to clarify certain decisions.

[alexwo]

/retest

[guydc]

If all pods loose connectivity, don't we expect that they would all fail eventually once becoming leaders?

[alexwo]

Pods rely on connectivity to the API server to participate in leader election processes. Without this connectivity, they cannot become leaders, in this test we won't bring it back. The secondary pods will continue to operate using their last known good configuration (xDS) and share it with envoy proxies accordingly.

[zhaohuabing]

@alexwo Could you please add a few lines of the comments for each test? (Like you explained in the comments of this PR) This will help other contributors understand why we need the test and how the test works. Thanks!

[arkodg]

#4767 has been merged, that should unblock this PR now ?

[zhaohuabing]

#4767 has been merged, that should unblock this PR now ?

@arkodg I believe this PR is ready to be merged.

Should we also cherry-pick this to v1.2.4?

[alexwo]

/retest