[Microsoft.Extensions.Http.Resilience] Questions on standard hedging handler

[lvxiaoxin]

Hi all,

I have a question about how hedging strategy and circuit breaker strategy hook up in the standard hedging handler.

I noticed that in the Standard hedging handler defaults, the doc is saying The standard hedging uses a pool of circuit breakers to ensure that unhealthy endpoints aren't hedged against. However, I looked through the code of AddStandardHedgingHandler, the action generator it passed to hedging strategy which used to select the next available routing endpoint is simply doing things like index++ without taking the circuit breaker state into consideration, and I doesn't find where the circuit breaker pool it implemented. I also checked the circuit breaker source code in Polly and seems it doesn't provide way for others to check the current circuit state. So I don't get how the standard hedging handler could achieve the goal to avoid hedging unhealthy endpoint.

To my understanding of the code reading, let's say there are 2 endpoints configured to the hedging strategy and the first one is cut by the circuit breaker. Then for every request, the code flow will be aways try the first endpoint and fail on the circuit breaker open exception, and then switch to the second endpoint by hedging. It cannot skip the first circuit broken endpoint, and will always trigger OnHedging event.

Am I correct?

[lvxiaoxin]

I debugged step by step and could confirm that, when circuit breaker of the primary endpoint group is open and the primary endpoint is unhealthy, the request will still always try the first group, fail on circuit breaker exception and then reach the secondary endpoint group. So I'm wondering it's just the words in doc here not that clear and lead to a few confusion.

https://learn.microsoft.com/en-us/dotnet/core/resilience/http-resilience?tabs=dotnet-cli#standard-hedging-handler-defaults

[iliar-turdushev]

Hello @lvxiaoxin,

when circuit breaker of the primary endpoint group is open and the primary endpoint is unhealthy, the request will still always try the first group, fail on circuit breaker exception and then reach the secondary endpoint group.

The behavior you described is exactly how the hedging handler is configured and works. We do use a pool of circuit breakers: one circuit breaker per URL authority. The following line of code configures the hedging handler to use a dedicated circuit breaker per URL authority of a hedged request. This is what we called "a pool of circuit breakers". This helps us to ensure that (1) hedged requests to unhealthy endpoints are blocked AND (2) circuit breakers of unhealthy endpoints don't block execution of hedged requests to other endpoints.

While the OnHedging event occurs the actual request is not sent to the endpoint. What behavior did you expect?

Will it be more clear if we rephrase our documentation as following: "The standard hedging uses a pool of circuit breakers to ensure that hedged requests to unhealthy endpoints are blocked."?

[lvxiaoxin]

Thanks @iliar-turdushev yes that documentation update will be much better. The existing words "aren't hedged against" sounds like even won't get a try on that routing flow path :) ~