Multiplexing gRPC & HTTP handlers with H2C+Prior Knowledge

[remko]

Background: Services on Cloud Run can only expose 1 port. If you want to expose a combination of gRPC and a regular REST API, you have to multiplex them. (e.g. using cmux)

In a standard setup, Cloud Run downgrades HTTP/2 requests to HTTP/1, and the setup with cmux seems works fine.
However, if you want to support gRPC streaming, you have to enable End-to-end HTTP/2 support on your Cloud Run service. The Cloud Run instructions say:

Your Cloud Run service must handle requests in HTTP/2 cleartext (h2c) format, because TLS is still terminated automatically by Cloud Run. To confirm that your service supports h2c requests, test the service locally using this cURL command: curl -i --http2-prior-knowledge http://localhost:PORT

So, my current setup is:

gsrv := grpc.NewServer(...)
hsrv := &http.Server{ Handler: h2c.NewHandler(..., &http2.Server{})}
srv := cmux.New(conn)
go gsrv.Serve(srv.MatchWithWriters(cmux.HTTP2MatchHeaderFieldSendSettings("content-type", "application/grpc")))
go hsrv.Serve(srv.Match(cmux.Any()))

This unfortunately does not work (even locally, no Cloud Run involved):

• Without the gRPC matcher, normal REST (i.e. hsrv, using curl --http2-prior-knowledge) works fine
• Once I add the gRPC matcher, gRPC works, but normal REST (using curl --http2-prior-knowledge) no longer works: I get the error http2: server connection error from [::1]:64054: connection error: PROTOCOL_ERROR

Without knowing the code or HTTP/2, my guess from what I'm seeing is that the cmux matcher sends a http2 settings frame during negotiation, but when it still fails to match gRPC and falls back to the next 'regular' http2 handler, the http2 handler starts negotiation from scratch and sends its own settings frame, which causes problems

[howardjohn]

I would recommend reading https://ahmet.im/blog/grpc-http-mux-go/, I think it addresses this

[remko]

Hi @howardjohn . Thanks for the link. However, I wanted to avoid using ServeHTTP because it is experimental and not officially supported, and has performance issues.

[howardjohn]

I don't think cmux can handle this. Since its a connection muxer. cloudrun (and Istio, FWIW) will multiplex gRPC and plain http2 methods onto the same connection. So if you use cmux, you might be able to get it to look like it works, but start seeing obscure failures in real world when the connections are reused. That blog was the result of us finding out that problem in prod 😛

[lukeo3o1]

Without knowing the code or HTTP/2, my guess from what I'm seeing is that the cmux matcher sends a http2 settings frame during negotiation, but when it still fails to match gRPC and falls back to the next 'regular' http2 handler, the http2 handler starts negotiation from scratch and sends its own settings frame, which causes problems

https://github.com/lukeo3o1/cmux/blob/677b4b20e70ac112b035428e88c80dd673af9b59/matchers.go#L252-L289

Maybe fix your guess