Add custom behavior to Tonic Connections

[giladwolff]

Feature Request

Expose a way to add a custom behavior to a connection.

Crates

tonic

Motivation

Tonic instantiates a service stack when creating a connection (see here). Some of the layers in the service stack are dynamic and generated as a result of configuration contained in the Endpoint object. There is no way to add an arbitrary layer to this stack to modify the behavior of a connection.

We use a balance_channel to manage a list of endpoints for our grpc clients. We
have two use-cases that will benefit from the ability to add custom behavior:
(a) adding circuit-breaker per endpoint and (b) adding metrics per connection. On top of that, adding this can help with other use cases like #1970.

Proposal

I can think of two ways to add this behavior. One is to add a custom layer as part of
the connection service instantiation here. Another is to add a custom layer when instantiating a service as part of
MakeSendRequestService.

I will be happy to work on a solution - I played with both of the above but
before investing more time would like to agree upon a solution.

Alternatives

One way to overcome this is to instantiate multiple grpc clients each one
using a single endpoint channel. I will probably go down that route if
there is no other way.