Add a resilience test suite to cover the expected behavior of the data plane for both existing and new proxy instances during specific edge conditions

[alexwo]

Description:
Consider having a resilience test suite test to cover the expected behavior of the data plane for both existing and new proxy instances during specific edge conditions—such as API server unavailability, Envoy Gateway (EG) downtime, or EG leader failures.

1. Ensuring Continuous Operation:
   By testing scenarios where the control plane components are unavailable, we can verify that existing and new proxy instances maintain their functionality.

2. Validating Last Known Good (LKG) State Persistence:
   Assessing how long the LKG state persists in active EG/Envoy instances during disruptions.

3. Preventing Premature Readiness of Unconfigured Instances:
   Ensuring that unconfigured Envoys/EGs do not reach a ready state prematurely prevents potential misrouting or service failures, thereby maintaining the integrity of the data plane.

4. Reflecting Real-World Scenarios:
   Running EG in a production-like setting with leader election enabled and multiple instances ensures that the tests accurately represent real-world operations, providing confidence in the system’s resilience when things go wrong.

[alexwo]

we can also add a test case to cover #4845 (comment) once fixed

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days.

[arkodg]

can this be closed @alexwo ?