Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

#292 Re-enable ReflectToSite and upgrade utls to v1.6.7 #633

Merged
merged 5 commits into from
Nov 22, 2024

Conversation

Jovis7
Copy link
Contributor

@Jovis7 Jovis7 commented Nov 20, 2024

@hwh33
Copy link
Contributor

hwh33 commented Nov 20, 2024

I just wanted to document your fantastic testing process here (copy/pasted from the linked Google Doc):

[Testing Steps]

  1. Create http-proxy branch “jovis-issue292-upgrade-utls-reenable-reflect-to-site” from “main”
  2. Create docket image & test track “arbok” & route ID
./bin/lc routes new --count 1 --provider OCI --location eu-frankfurt-1 --region eu1 --track arbok
  1. Run lantern-client (main branch updated Nov 20) against route ID
    → Test Result : Get frankfurt IP and browse internet.
  2. Delete docket image & track & route
  3. Modify http-proxy : Re-enable reflect-to-site (by setting continueOnError to false. Revert this PR)
    Commit here
  4. Repeat step 2 ~ 4 for testing
    → Test Result : lantern-client failed to get IP.
    This means there is an issue if reflect-to-site enabled.
  5. Modify http-proxy : Upgrade utls to v1.6.7
    Commit here
  6. Repeat step 2 ~ 4 for testing
    → Test Result : Get frankfurt IP and browse internet
    This means it’s working with new utsl + reflect-to-site enabled.

Copy link
Contributor

@hwh33 hwh33 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one minor thing and it was due to me changing my mind 🙂 . Once that's changed, this is good to go! Excellent investigation and work on this fix.

Let's be careful about merging this. Once this is merged, the proxies will pick up this change. Your test indicates that this should be fine since clients in the field should not need an update. However, we should watch traffic to reflect-to-site proxies just in case.

So, let's do the following:

  1. Please make the suggested change to helloError.
  2. I will merge this PR one day in the morning my time.
  3. I will monitor the metrics to ensure this change does not have negative effects for our users. If I see any negative effects, I will revert the merge and let you know.
  4. I will document here the metrics that I watch for your future reference.

@@ -205,16 +205,16 @@ func (rrc *clientHelloRecordingConn) processHello(info *tls.ClientHelloInfo) (*t
// pre-defined tickets. If it doesn't we should again return some sort of error or just
// close the connection.
if !helloMsg.TicketSupported {
return rrc.helloError("ClientHello does not support session tickets", true)
return rrc.helloError("ClientHello does not support session tickets", false)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since continueOnError is always set to false now, let's just remove that parameter from clientHelloRecordingConn.helloError. Sorry, I think that's different than what I originally told you!

@Jovis7
Copy link
Contributor Author

Jovis7 commented Nov 21, 2024

@hwh33 Removed continueOnError and thanks for the review!
Today I've tested this branch on "track" again, and will see how to monitor the metrics once you document here.

@hwh33
Copy link
Contributor

hwh33 commented Nov 21, 2024

Awesome @Jovis7, thank you! I will merge this tomorrow morning my time and update this PR with the results.

@hwh33
Copy link
Contributor

hwh33 commented Nov 22, 2024

Merging this now! As noted above, this should trigger an automatic update of our production proxies. I'll verify that this update re-enables reflect-to-site by trying a TLS connection to one of our TLS tracks (note that currently all TLS tracks run reflect-to-site). This should get "reflected" to the masquerade site and I should receive the certificate from that site. Currently, I see the proxy's certificate instead of the masquerade's certificate (as expected since reflect-to-site is currently disabled):

Route Details

(this is just a random HTTPs route I grabbed)

> bin/lc route dump-config 160c283e-219d-4170-ad5d-57a844394583
{
  "addr": "79.72.66.235",
  "track": "jigglypuff",
  "location": {
    "city": "Frankfurt",
    "country": "Germany",
    "countryCode": "DE",
    "latitude": 50.11,
    "longitude": 8.68
  },
  "name": "jigglypuff-160c283e-219d-4170-ad5d-57a844394583",
  "port": 443,
  "protocol": "tls",
  "certPem": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJ6RENDQVhLZ0F3SUJBZ0lRUEJMZlF3Y3dvdWpVZlIrcTFZMS9GekFLQmdncWhrak9QUVFEQWpCRU1SSXcKRUFZRFZRUUtFd2x0WVdOb2FXRjJaV3d4TGpBc0JnTlZCQU1USldGdWRHbHpZMjlzYVdNZ2RXNTNhWE5sYm1WegpjeWR6SUc1dmJtRnRZblZzWVhSdmNua3dIaGNOTWpReE1ETXhNakV4TkRJd1doY05NalV4TURNeE1qRXhOREl3CldqQkVNUkl3RUFZRFZRUUtFd2x0WVdOb2FXRjJaV3d4TGpBc0JnTlZCQU1USldGdWRHbHpZMjlzYVdNZ2RXNTMKYVhObGJtVnpjeWR6SUc1dmJtRnRZblZzWVhSdmNua3dXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQwpBQVI5eTl5Y1lFUjlSVkJxdkV4b0l0U2Y2czNFNU9TWGxBZWp3dTJldzRHc0VJKzk2QkxuVjdPUFJNTURINWFjCk5uTmt2NmlaU1JvdmFTVnlLaFZ2a0l2dW8wWXdSREFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3cKQ2dZSUt3WUJCUVVIQXdFd0RBWURWUjBUQVFIL0JBSXdBREFQQmdOVkhSRUVDREFHaHdSUFNFTHJNQW9HQ0NxRwpTTTQ5QkFNQ0EwZ0FNRVVDSVFDT25nMGdsSFRBdHlBNXFwYUN4azJiVGdMYVY1NUF5RWpIUVd1UG5NOGV0QUlnCk9XRTFERVdJUG5ITXp1UFlJb29HUThUWExVQld4Y1RVVW9jMjJvWkpkcEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",
  "authToken": "c2b186f8d7df2f74490c44608b3dac85b09070e963b1576846d5a48cf7b7b41e",
  "trusted": true,
  "connectCfgTls": {
    "sessionState": {
      "sessionTicket": "Z9s76HOvDBfWZ8DTw68yQjF9mxtupCHZTs5uTMkxfr7NuhOw/mDtXr+UobuC5Or0Cs/mdTRsJ6l/KnziE6LBewndm2US26cp1GKVqSQuC3ZpEgeLPMj99s/Bg7c5J84uP5YzgnICLBIX2fJAeMicEwvDLLoyqAUUPA==",
      "version": 771,
      "cipherSuite": 49195,
      "masterSecret": "maQFgJrDMrwKUJn/KKhSYM1EFbdpTe7hpWY3PUevxIwivw+L50c5fCajsvYncX6m"
    },
    "serverNameIndicator": "ukr.net"
  }
}
Test TLS Connection
> openssl s_client -connect 79.72.66.235:443 -servername ukr.net
Connecting to 79.72.66.235
CONNECTED(00000003)
depth=0 O=machiavel, CN=antiscolic unwiseness's nonambulatory
verify error:num=18:self-signed certificate
verify return:1
depth=0 O=machiavel, CN=antiscolic unwiseness's nonambulatory
verify return:1
---
Certificate chain
 0 s:O=machiavel, CN=antiscolic unwiseness's nonambulatory
   i:O=machiavel, CN=antiscolic unwiseness's nonambulatory
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: ecdsa-with-SHA256
   v:NotBefore: Oct 31 21:14:20 2024 GMT; NotAfter: Oct 31 21:14:20 2025 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIBzDCCAXKgAwIBAgIQPBLfQwcwoujUfR+q1Y1/FzAKBggqhkjOPQQDAjBEMRIw
EAYDVQQKEwltYWNoaWF2ZWwxLjAsBgNVBAMTJWFudGlzY29saWMgdW53aXNlbmVz
cydzIG5vbmFtYnVsYXRvcnkwHhcNMjQxMDMxMjExNDIwWhcNMjUxMDMxMjExNDIw
WjBEMRIwEAYDVQQKEwltYWNoaWF2ZWwxLjAsBgNVBAMTJWFudGlzY29saWMgdW53
aXNlbmVzcydzIG5vbmFtYnVsYXRvcnkwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNC
AAR9y9ycYER9RVBqvExoItSf6s3E5OSXlAejwu2ew4GsEI+96BLnV7OPRMMDH5ac
NnNkv6iZSRovaSVyKhVvkIvuo0YwRDAOBgNVHQ8BAf8EBAMCBaAwEwYDVR0lBAww
CgYIKwYBBQUHAwEwDAYDVR0TAQH/BAIwADAPBgNVHREECDAGhwRPSELrMAoGCCqG
SM49BAMCA0gAMEUCIQCOng0glHTAtyA5qpaCxk2bTgLaV55AyEjHQWuPnM8etAIg
OWE1DEWIPnHMzuPYIooGQ8TXLUBWxcTUUoc22oZJdpI=
-----END CERTIFICATE-----
subject=O=machiavel, CN=antiscolic unwiseness's nonambulatory
issuer=O=machiavel, CN=antiscolic unwiseness's nonambulatory
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 864 bytes and written 415 bytes
Verification error: self-signed certificate
---
New, TLSv1.2, Cipher is ECDHE-ECDSA-AES128-GCM-SHA256
Server public key is 256 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-ECDSA-AES128-GCM-SHA256
    Session-ID: 0F38CB352E08AF1BAAB5EAA2AC7D2C3BC26B20E683C66152D113625138F4E2E2
    Session-ID-ctx:
    Master-Key: 47BB161113A65FB007C9DDB16141E347E70175393F933EFB51C306133EC73C3B7F20A7DA4F3B98FB42948A59926889D7
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket:
    0000 - 2d c5 f5 f0 50 a3 09 66-23 42 53 7a dd 5b 84 9e   -...P..f#BSz.[..
    0010 - 7c 5c ca 24 97 b4 3e 95-6a a2 fa 35 3b 70 83 20   |\.$..>.j..5;p.
    0020 - a8 fb fb f0 60 8c 75 9f-46 11 de bd 25 69 1a e7   ....`.u.F...%i..
    0030 - 11 69 6b da 5b cd 89 53-39 36 d8 e8 12 ba 40 7d   .ik.[..S96....@}
    0040 - 87 73 8d 35 ee 15 8e 78-25 e1 63 41 03 42 40 cb   .s.5...x%.cA.B@.
    0050 - 22 9f f1 6e a8 08 c6 42-94 8f 86 d2 16 a4 8f 48   "..n...B.......H
    0060 - 29 e3 87 28 5e 2b d2 f1-ec 4c aa f1 8d 5c 36 a8   )..(^+...L...\6.
    0070 - 30 13 57 9a 05 2c c9 4a-50                        0.W..,.JP

    Start Time: 1732298699
    Timeout   : 7200 (sec)
    Verify return code: 18 (self-signed certificate)
    Extended master secret: yes
---

I will monitor performance of our reflect-to-site tracks by watching our track dashboard in Signoz (shout out to @Crosse for setting that dashboard up!) and filtering for https_multiplex (again note that all TLS / HTTPs tracks run reflect-to-site). The two graphs I'll be watching most closely are "Total Throughput vs Last Week" and "Connections". If clients are unable to connect with their configured session tickets, we will see these metrics drop off. Hopefully we see no changes!

@hwh33 hwh33 merged commit 5e9b06c into main Nov 22, 2024
1 check passed
@hwh33
Copy link
Contributor

hwh33 commented Nov 22, 2024

I am now receiving the masquerade certificate, indicating that reflect-to-site is re-enabled as intended. No change in the metrics so far, but I think it's a little early to say for sure. I'll keep watching!

Test TLS Connection
openssl s_client -connect 79.72.66.235:443 -servername ukr.net
Connecting to 79.72.66.235
CONNECTED(00000003)
depth=2 C=US, O=Google Trust Services LLC, CN=GTS Root R1
verify return:1
depth=1 C=US, O=Google Trust Services, CN=WR3
verify return:1
depth=0 CN=ukr.net
verify return:1
---
Certificate chain
 0 s:CN=ukr.net
   i:C=US, O=Google Trust Services, CN=WR3
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Nov 17 04:18:50 2024 GMT; NotAfter: Feb 15 05:14:45 2025 GMT
 1 s:C=US, O=Google Trust Services, CN=WR3
   i:C=US, O=Google Trust Services LLC, CN=GTS Root R1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Dec 13 09:00:00 2023 GMT; NotAfter: Feb 20 14:00:00 2029 GMT
 2 s:C=US, O=Google Trust Services LLC, CN=GTS Root R1
   i:C=BE, O=GlobalSign nv-sa, OU=Root CA, CN=GlobalSign Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 19 00:00:42 2020 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFJjCCBA6gAwIBAgIRAKildecQ5sBIEh5jyZ9X8r8wDQYJKoZIhvcNAQELBQAw
OzELMAkGA1UEBhMCVVMxHjAcBgNVBAoTFUdvb2dsZSBUcnVzdCBTZXJ2aWNlczEM
MAoGA1UEAxMDV1IzMB4XDTI0MTExNzA0MTg1MFoXDTI1MDIxNTA1MTQ0NVowEjEQ
MA4GA1UEAxMHdWtyLm5ldDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
ANbYXplLasTQeWMLZ9ckb5rWaIvnGMKZKNGG4azQJbGRtvFCEWkdkA9vN207Iuf4
U+3Hgun5MyHk0X0qYI2hOyZg1xNO91kE0Go+ZStBTWZmTjcRvWTZ0EhlAixpOQr3
AgHzoiG6C1hv/ypuo+Wa7F/rzfq+K3Mm0hIeb1JyM9SGXEmIZLSMC5rGXWN386pk
oL0SYGW2l6choGY8rHtK7k9qfnUKPHeOrqRPwwmqaM9VEslEYBQ+W3t81VUTDwCB
apnbBCM5aaKvjeT2cRzmJ8U+hVomGLScf2NmFmiAUHdsMaIcBBx1SS39lZKdpz/h
sWxlibdQzXbZzNQMeSJHi2UCAwEAAaOCAkwwggJIMA4GA1UdDwEB/wQEAwIFoDAT
BgNVHSUEDDAKBggrBgEFBQcDATAMBgNVHRMBAf8EAjAAMB0GA1UdDgQWBBTAh+G+
q1AywYciCNXPF33591ZX5jAfBgNVHSMEGDAWgBTHgfX9jojZADxNY6JQMSSgziP+
IzBeBggrBgEFBQcBAQRSMFAwJwYIKwYBBQUHMAGGG2h0dHA6Ly9vLnBraS5nb29n
L3Mvd3IzL3FLVTAlBggrBgEFBQcwAoYZaHR0cDovL2kucGtpLmdvb2cvd3IzLmNy
dDAdBgNVHREEFjAUggd1a3IubmV0ggkqLnVrci5uZXQwEwYDVR0gBAwwCjAIBgZn
gQwBAgEwNgYDVR0fBC8wLTAroCmgJ4YlaHR0cDovL2MucGtpLmdvb2cvd3IzL05W
NU9RQnJ0RDlzLmNybDCCAQUGCisGAQQB1nkCBAIEgfYEgfMA8QB3AE51oydcmhDD
OFts1N8/Uusd8OCOG41pwLH6ZLFimjnfAAABkziPOG4AAAQDAEgwRgIhANEofv22
pIW1nOuEd0IyOGrucAgoMu/Foho2TLWoKMI+AiEAtPsDqmgq8X7IgQQbnXnqPved
hosuh7ZzLyST46Ilq6kAdgDm0jFjQHeMwRBBBtdxuc7B0kD2loSG+7qHMh39HjeO
UAAAAZM4jzxrAAAEAwBHMEUCIHi+kipDwKUC9pCalIO0p33CVPiKJjO/ZaGym2/n
KEn8AiEA+VoPERG0r7ZjwZlSZ6EVk3Nwt0Q+AJHEbWY6KZpkqhEwDQYJKoZIhvcN
AQELBQADggEBAFpopEOclVkZZQqrdREYybiFNInbuTEtvBThOZg8v1HARLPRhiyZ
8n7ZKEMRQ8FzXKopcwq74+b/D0EVtjFob9W0RwLTV1Cy7ZmFxPyY8kPy6/tpt7cc
mzoVtSWps9K4EvTxYN0SHkkzxNxl3UkeEVHnGUy4Qmb/jU8zr2MriAQpUFIdAONr
YP/rajtVkL+MchyhRjz22FCFKUaENEMgOqEnXQOZ8zDOQ5kzbbHbUHSUzF/5ptLX
q37Sagmw4Mk6j0XC09Q2RM9C8jYi5sw9XoLX6K3LUj3fGIUSbFbNWT0Fv9Kpj+fr
yJzc8ooQSPhKsprRI1i39J0mH85BfwyOi20=
-----END CERTIFICATE-----
subject=CN=ukr.net
issuer=C=US, O=Google Trust Services, CN=WR3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4499 bytes and written 402 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
This TLS version forbids renegotiation.
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

@hwh33
Copy link
Contributor

hwh33 commented Nov 22, 2024

I was able to successfully connect to the TLS proxy I tested previously with a Lantern Client, running the following in lantern-client:

LANTERN_CLOUD=$GL/lantern-cloud ./hit_proxy.bash 160c283e-219d-4170-ad5d-57a844394583

I verified (via the logs) that my client was using the configured proxy and I was able to browse YouTube and other sites.

@Crosse Crosse deleted the jovis-issue292-upgrade-utls-reenable-reflect-to-site branch November 22, 2024 21:50
@hwh33
Copy link
Contributor

hwh33 commented Nov 22, 2024

Metrics are all still looking hunky dory. I think you've done it @Jovis7! Excellent work on this 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants