Fix: frequent 503 errors when connecting to a Service experiencing high Pod churn

[zhaohuabing]

Reverts #4337
Fixes: #4685

Release Note: Yes

To avoid regression of the bug fixed by #4337, a temporary buffer has been added to store events received before the Updater is enabled.
These events will be sent to the update channel once the Updater is enabled.

Tested the fix in my local env:

➜  k -n envoy-gateway-system logs envoy-gateway-7f7d68bd-q74m6 |grep "status update"                        
2024-11-26T05:56:03.461Z        INFO    provider        kubernetes/status_updater.go:185        received a status update while disabled, storing for later      {"runner": "provider", "event": {
"name":"ha-gateway","namespace":"gateway-upgrade-infra"}}
2024-11-26T05:56:03.764Z        INFO    provider        kubernetes/status_updater.go:185        received a status update while disabled, storing for later      {"runner": "provider", "event": {
"name":"upgrade"}}                                                                              
2024-11-26T05:56:03.767Z        INFO    provider        kubernetes/status_updater.go:185        received a status update while disabled, storing for later      {"runner": "provider", "event": {
"name":"http-backend-eg-upgrade","namespace":"gateway-upgrade-infra"}}
2024-11-26T05:56:03.767Z        INFO    provider        kubernetes/status_updater.go:185        received a status update while disabled, storing for later      {"runner": "provider", "event": {
"name":"ha-gateway","namespace":"gateway-upgrade-infra"}}
2024-11-26T05:56:28.146Z        INFO    provider        kubernetes/status_updater.go:139        started status update handler   {"runner": "provider"}
2024-11-26T05:56:28.146Z        INFO    provider        kubernetes/status_updater.go:197        sending stored status update    {"runner": "provider", "event": {"name":"ha-gateway","namespace":
"gateway-upgrade-infra"}}       
2024-11-26T05:56:28.146Z        INFO    provider        kubernetes/status_updater.go:197        sending stored status update    {"runner": "provider", "event": {"name":"upgrade"}}
2024-11-26T05:56:28.146Z        INFO    provider        kubernetes/status_updater.go:197        sending stored status update    {"runner": "provider", "event": {"name":"http-backend-eg-upgrade"
,"namespace":"gateway-upgrade-infra"}}  
2024-11-26T05:56:28.146Z        INFO    provider        kubernetes/status_updater.go:197        sending stored status update    {"runner": "provider", "event": {"name":"ha-gateway","namespace":
"gateway-upgrade-infra"}}       

If anyone wants to verify this PR, you can do so by using this image zhaohuabing/gateway-dev:52e6d8577

[codecov]

Codecov Report

Attention: Patch coverage is 60.60606% with 13 lines in your changes missing coverage. Please review.

Project coverage is 65.58%. Comparing base (48a0310) to head (3b9ba84).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/provider/kubernetes/status_updater.go	60.60%	12 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4754      +/-   ##
==========================================
- Coverage   65.60%   65.58%   -0.02%     
==========================================
  Files         211      211              
  Lines       31961    31989      +28     
==========================================
+ Hits        20968    20981      +13     
- Misses       9753     9766      +13     
- Partials     1240     1242       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zirain]

should we change the PR title?

[arkodg]

should we change the PR title?

yeah +1, because this PR does more than just revert

[zirain]

/retest

[zirain]

follow up: need a test to make sure refactor won't broken again.