feat(transport): TLS StreamDialer & Dynamic HTTP CONNECT #117

fortuna · 2023-10-25T17:57:56Z

This PR introduces the TLS transport. It also adds a way to dynamically specify a transport in the local proxy HTTP CONNECT.

This will help us explore domain-fronting, DNS-over-TLS and SOCKS5-over-TLS, and address SNI-based censorship.

Protocols over TLS

This works for DNS-over-TLS:

go run github.com/Jigsaw-Code/outline-sdk/x/examples/resolve -tcp -transport "split:2|tls:sni=decoy.com" -type a -resolver dns.google:853 www.example.com

Given a SOCKS5 server [host:port], you can do SOCKS5-over-TLS with:

go run github.com/Jigsaw-Code/outline-sdk/x/examples/fetch  -transport "tls|socks5:[host:port]"  http://www.rferl.org:443/

Domain Fronting

This now works for domain-fronting:

go run github.com/Jigsaw-Code/outline-sdk/x/examples/fetch  -transport "tls:sni=www.example.com"  -v http://www.rferl.org:443/

The example below shows the the bypass censorship of RFE/RL in a network Russia.

Running a local proxy (KEY is a config for a server in Russia):

go run github.com/Jigsaw-Code/outline-sdk/x/examples/http2transport -transport "$KEY|tls:sni=www.example.com&certName=www.rferl.org" -localAddr localhost:8080

Fetching the page via the proxy in Russia and domain-fronting:

curl -p -x http://localhost:8080 --connect-to ::e4887.dscb.akamaiedge.net: http://www.rferl.org:443/ -D -

Note that I had to tell Curl to connect to e4887.dscb.akamaiedge.net in order to bypass DNS-based blocking. That forces the need to specify the certName=www.rferl.org in the transport, otherwise is tries to use the dialed address (e4887.dscb.akamaiedge.net).

This is not ideal. In the future I'd like to have a transport config that will map the domain to resolve and connect to. That way the user can keep dialing the intended destination. An example with a fictitious fixed dialer that always dials the same address could look like:

"fixed:e4887.dscb.akamaiedge.net|tls:sni=www.example.com"

We may want to introduce matchers based on host:port, do a allow a dialer to be used across endpoints. An example where we map www.rferl.org to e4887.dscb.akamaiedge.net, and only changes the SNI for TLS connection could look like:

"[www.rferl.org:*]fixed:e4887.dscb.akamaiedge.net|[www.rferl.org:443]tls:sni=www.example.com"

Dynamic HTTP CONNECT

With the HTTP CONNECT proxy, we can now use the Transport header to specify a transport config that wraps the base dialer. So a tunnel request could look like:

CONNECT www.rferl.org:443
Transport: tls:sni=example.com

or

CONNECT www.rferl.org:443
Transport: ss://[OUTLINE_KEY]

This allows the client to dynamically change the transport on a per-connection basis, so the transport selection logic can live in Flutter, for example. This also mitigates the issue of the transport config not working for every endpoint.

Below is the domain-fronting example above, modified to use dynamic transports.

Proxy (just connects to the Russian proxy, no bypass logic):

go run github.com/Jigsaw-Code/outline-sdk/x/examples/http2transport -transport "$KEY" -localAddr localhost:8080

Request with domain-fronting:

curl -p -x http://localhost:8080 --proxy-header 'Transport: tls:sni=www.example.com&certName=www.rferl.org' --connect-to ::e4887.dscb.akamaiedge.net: http://www.rferl.org:443/ -D -

jyyi1

Thanks! Suggested PR title: "feat(transport): TLS StreamDialer"

transport/tls/tls.go

transport/tls/stream_dialer.go

fortuna · 2023-10-25T19:52:59Z

To add to this PR: reuse https://pkg.go.dev/crypto/tls#ClientSessionCache

transport/tls/stream_dialer.go

fortuna · 2023-10-26T17:30:20Z

Hold on a bit. I want to move TLS to x/ and experiment with some new approaches.
We probably need an api where we can configure on a per-connection basis.
The goal being so that we can pass configs to the mobileproxy and let the user pick the strategies that make sense.

jyyi1

Thanks! The code looks good, added some comments to double check.

README.md

x/config/config.go

jyyi1 · 2023-10-27T21:55:28Z

x/tls/stream_dialer.go

+
+var _ transport.StreamConn = (*streamConn)(nil)
+
+func (c streamConn) CloseRead() error {


Do we need to override CloseWrite to close both connections as well?

I believe tls.Conn already does that.

Yes, but it won't call innerConn.CloseWrite(), just wanna make sure this is expected.

... and does not call CloseWrite on the underlying connection ...

I think you are right. I added the call to close the inner conn

x/config/config.go

x/httpproxy/connect_handler.go

jyyi1 · 2023-10-27T22:09:11Z

x/httpproxy/connect_handler.go

@@ -48,9 +75,16 @@ func (h *connectHandler) ServeHTTP(proxyResp http.ResponseWriter, proxyReq *http
 	}

 	// Dial the target.
-	targetConn, err := h.dialer.Dial(proxyReq.Context(), proxyReq.Host)
+	transportConfig := proxyReq.Header.Get("Transport")


Shall we prefix the header name with sth like X-Outline-Transport?

That goes against the official current best practice: https://www.rfc-editor.org/rfc/rfc6648

We could consider Outline-Transport though. It's longer, more things to type when I write command lines, but it could make it clear it's our header, in case we add multiple ones.

Co-authored-by: J. Yi <[email protected]>

fortuna · 2023-10-30T17:59:40Z

FYI, I fixed the flaky test, because it was blocking my PR

jyyi1

LGTM, I think we just need to double confirm the CloseWrite behavior is expected (replied in the comment above).

fortuna requested a review from jyyi1 October 25, 2023 17:57

jyyi1 reviewed Oct 25, 2023

View reviewed changes

transport/tls/tls.go Outdated Show resolved Hide resolved

transport/tls/stream_dialer.go Outdated Show resolved Hide resolved

transport/tls/stream_dialer.go Outdated Show resolved Hide resolved

transport/tls/stream_dialer.go Outdated Show resolved Hide resolved

fortuna changed the title ~~Create TLS transport~~ feat(transport): TLS StreamDialer Oct 26, 2023

github-advanced-security bot found potential problems Oct 26, 2023

View reviewed changes

transport/tls/stream_dialer.go Fixed Show fixed Hide fixed

fortuna requested a review from jyyi1 October 26, 2023 17:02

fortuna marked this pull request as draft October 26, 2023 17:28

fortuna removed the request for review from jyyi1 October 26, 2023 17:30

fortuna added 14 commits October 27, 2023 09:26

Create TLS transport

1fd2df3

Address review

a67908c

Fix test

09ae7a1

Fix Dial

27fa875

Add Test

a6994f7

Add NewStreamDialer

554feca

Add support for domain-fronting

948c53f

Add tests

118a35b

Make config public

0061ab1

Update README

45aa484

Improve selection

68d773d

Fix links

c00c262

Move tls to x/

b80e484

Add tls to config.

0e9afd8

fortuna force-pushed the fortuna/tls branch from 2eef1f5 to 0e9afd8 Compare October 27, 2023 13:27

fortuna added 4 commits October 27, 2023 09:58

Add transport header to connect handler.

df64eba

Add comment

79dd0d8

Revamp TLS config

c79e3cd

Add handler change

422bcdc

fortuna marked this pull request as ready for review October 27, 2023 17:03

fortuna requested a review from jyyi1 October 27, 2023 17:07

fortuna added 7 commits October 27, 2023 14:02

Cleanup

125a1fd

Add test

59b4bd3

Add test for combined

3ec98c6

Better errors

2ab8917

Better request cancelled error

9150f90

Add config doc

f2be501

Sanitize and output errors

e0dcc03

jyyi1 reviewed Oct 27, 2023

View reviewed changes

fortuna and others added 4 commits October 30, 2023 10:51

Update README.md

d443793

Co-authored-by: J. Yi <[email protected]>

Update stream split readme

6b23ada

Address review comments

f238e12

Fix test

1fb0a59

fortuna requested a review from jyyi1 October 30, 2023 17:41

jyyi1 approved these changes Oct 30, 2023

View reviewed changes

Fix CloseWrite

9caadf5

fortuna requested a review from jyyi1 October 30, 2023 21:10

fortuna changed the title ~~feat(transport): TLS StreamDialer~~ feat(transport): TLS StreamDialer & Dynamic HTTP CONNECT Oct 30, 2023

fortuna merged commit c103980 into main Oct 30, 2023
6 checks passed

fortuna deleted the fortuna/tls branch October 30, 2023 22:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(transport): TLS StreamDialer & Dynamic HTTP CONNECT #117

feat(transport): TLS StreamDialer & Dynamic HTTP CONNECT #117

fortuna commented Oct 25, 2023 •

edited

Loading

jyyi1 left a comment •

edited

Loading

fortuna commented Oct 25, 2023

fortuna commented Oct 26, 2023

jyyi1 left a comment

jyyi1 Oct 27, 2023

fortuna Oct 30, 2023

jyyi1 Oct 30, 2023

fortuna Oct 30, 2023

jyyi1 Oct 27, 2023

fortuna Oct 30, 2023

fortuna commented Oct 30, 2023

jyyi1 left a comment


		var _ transport.StreamConn = (*streamConn)(nil)

		func (c streamConn) CloseRead() error {

feat(transport): TLS StreamDialer & Dynamic HTTP CONNECT #117

feat(transport): TLS StreamDialer & Dynamic HTTP CONNECT #117

Conversation

fortuna commented Oct 25, 2023 • edited Loading

Protocols over TLS

Domain Fronting

Dynamic HTTP CONNECT

jyyi1 left a comment • edited Loading

Choose a reason for hiding this comment

fortuna commented Oct 25, 2023

fortuna commented Oct 26, 2023

jyyi1 left a comment

Choose a reason for hiding this comment

jyyi1 Oct 27, 2023

Choose a reason for hiding this comment

fortuna Oct 30, 2023

Choose a reason for hiding this comment

jyyi1 Oct 30, 2023

Choose a reason for hiding this comment

fortuna Oct 30, 2023

Choose a reason for hiding this comment

jyyi1 Oct 27, 2023

Choose a reason for hiding this comment

fortuna Oct 30, 2023

Choose a reason for hiding this comment

fortuna commented Oct 30, 2023

jyyi1 left a comment

Choose a reason for hiding this comment

fortuna commented Oct 25, 2023 •

edited

Loading

jyyi1 left a comment •

edited

Loading