remove Transfer-Encoding from Proxy responses

[bentsku]

Motivation

We've had an issue with the Mailhog extension when using HTTP/1.1 (https://github.com/localstack/localstack-extensions/blob/main/mailhog), where the content would get double Transfer-Encoding: chunked header.

This was investigated, and we found out that Twisted would always automatically add the Transfer-Encoding: chunked header value if no content-length is set, even if there's already a value. Werkzeug has the same behavior, but hypercorn does not.

After investigation, PEP3333 (https://peps.python.org/pep-3333/#id34) actually says that a WSGI application should not set the header itself; as it is the responsibility of the web server to manage this. It is true that it felt wrong from the application itself to dictate the server behavior. I believe GZIP is a bit of a different beast, as it relates more to the content itself of the response than the way the server will send it back.

(Note: applications and middleware must not apply any kind of Transfer-Encoding to their output, such as chunking or gzipping; as “hop-by-hop” operations, these encodings are the province of the actual web server/gateway. See Other HTTP Features below, for more details.)

So the more natural fix is to remove the Transfer-Encoding: chunked value from the header returned by the proxy, as the webserver will take care of that. We have specific tests around the automatic adding of this header when the request does not have content-length and have a generator as response.

Changes

• add tests for Proxy and ProxyHandler behavior around transfer-encoding, asserting that the proxy itself does not have Transfer-Encoding in its response headers to proxy forward, and that the ProxyHandler properly returns chunked response
• selectively filter the Transfer-Encoding: chunked value out of Transfer-Encoding so that we do not return it, but keep gzip

\cc @lukqw

[bentsku]

this assertion (checking that the encoding is chunked) is somewhat testing the behavior of Werkzeug instead of the ProxyHandler itself, as it is the decision of Werkzeug to set the Transfer-Encoding to chunked if the data is a generator. I added the test more as documentation around the behavior and to put it to contrast the Proxy behavior itself

[thrau]

Really nice catch and 🕵️ work! The added complexity isn't great, but understand why it's needed. Header manipulation has caused us a lot of issues in the past, so let's keep a close eye on any impact it may have!