Need for HTTP proxy support

[ares]

In case a user wants to let the traffic flow through HTTP proxy, receptor should provide options to specify proxy url together with username and password.

[jhjaggars]

In order to support his we'll need to augment our ws_connect usage:

https://aiohttp.readthedocs.io/en/stable/client_reference.html#aiohttp.ClientSession.ws_connect

[ghjm]

Two questions for @ares:

1. Aiohttp explicitly doesn't support https proxies (Implement HTTPS Proxy Support aio-libs/aiohttp#3816). Do we need this or is it okay to support http proxies only?
2. Are you okay with setting an environment variable to http_proxy=http://username:password@proxy-server:port or do you want some other way of passing in the username/password?

[ares]

Ad 1. I'm afraid HTTPS is needed, is there a way to make it work with python 3.7?
Ad 2. I think the env variable is totally fine

[ghjm]

Aiohttp might support https proxies in a future version of Python 3.8. As of today they are not supported in any version.

[ghjm]

What we can do while still using aiohttp is done, in ansible/receptor#194. I have also opened ansible/receptor#197 for considering a switch to a different library.

Testing for this issue should verify that it works with HTTP_PROXY=http://<whatever> and HTTPS_PROXY=http://<whatever>. It is not expected to work when the specified proxy server is https. (Though it should give a meaningful error message in that case.)

[Ichimonji10]

I'm unfamiliar with HTTP proxies. What's the basic procedure for testing this? Is it to install, configure with credentials, and run Squid, and then verify that traffic is sent through it?

[ghjm]

Yes, that's basically correct @Ichimonji10. Squid is as good a proxy server to test with as any.

It might be nice to also verify that no_proxy works as expected. This is used when you need a node to make connections to both proxied and unproxied destinations (i.e. your proxy server can't proxy traffic to a destination inside the corporate network).

As an example of this, suppose you have a corporate network corp.com, with a proxy server at proxy.corp.com. You have three receptor nodes, receptor1.corp.com, receptor2.corp.com and receptor3.cloud.net. The first two are inside the corporate network, so trying to connect to them through the proxy server won't work. The third is at some cloud provider somewhere, and you need to use the proxy server to get to it.

This puts you in a catch-22 situation if you want receptor to make outbound connections to both receptor2 and receptor3 (and for some reason you want to use websockets for all of them). If you configure http_proxy=http://proxy.corp.com in the environment then you will be able to connect to ws://receptor3.cloud.net but your connection to ws://receptor2.corp.com will fail. If you don't configure https_proxy, then it will be the opposite.

So in this circumstance you configure http_proxy=http://proxy.corp.com and no_proxy=corp.com which ought to result in successfully connecting to both, with the external connection proxied and the internal connection not.

I'll leave it up to you how much of this you want to try to actually test.

[Ichimonji10]

Thank you, @ghjm! This is exactly what I need to know. Either @elyezer or I will get to this when we can. I'm mostly concerned with end-to-end testing of the app that builds on this (you know the one), but will get to this as I can.

[elyezer]

I am planning to tackle this earlier next week and all the information shared are really useful.

[ghjm]

Note: The fix for this is in ansible/receptor#194.

[elyezer]

This issue can be considered verified. The following was done to check it:

While running:

$ sudo tcpdump -aA -i lo -vvv "dst port 9999"

And

$ sudo tcpdump -aA -i lo -vvv "dst port 9998"

The following receptor process was run:

$ HTTP_PROXY=http://localhost:9998 poetry run receptor -d /tmp/ping ping node-a --peer ws://localhost:9999

Only the tcpdump listening to port 9998 events had an output.

Then the HTTP_PROXY variable was removed and the following command was run:

$ poetry run receptor -d /tmp/ping ping node-a --peer ws://localhost:9999

This time, only the tcpdump listening to port 9999 events had an output. Which brings us to the conclusion the proxy information is being used as expected.

Worth mentioning that receptor presented a stack trace with the aiohttp.client_exceptions.ClientProxyConnectionError exception while testing the proxy configuration and aiohttp.client_exceptions.ClientConnectorError when running without the proxy configuration. This is just to also confirm that the proxy support provided by the aiohttp lib is being used as expected.

[elyezer]

I've reopened this as I am trying to get it to work with a real proxy and I am not able. Please take into consideration that I don't know much about proxy configuration and I might be doing something wrong.

I tried to configure both nginx e squid and got similar issue:

aiohttp.client_exceptions.WSServerHandshakeError: 502, message='Invalid response status', url=URL('ws://localhost:9999')

To get that error I did the following:

Configured nginx to run as a Web Socket proxy:

http {
    map $http_upgrade $connection_upgrade {
        default upgrade;
        '' close;
    }
 
    upstream websocket {
        server localhost:9999;
    }
 
    server {
        listen 9990;
        location / {
            proxy_pass http://websocket;
            proxy_http_version 1.1;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Host $host;
        }
    }
}

Started a receptor node to listen on port 9999 as the following:

$ poetry run receptor --debug --node-id=controller -d /tmp/controller node --listen=ws://localhost:9999

Then tried to run another receptor node that would be using the proxy to access the node at port 9999:

$ http_proxy=http://127.0.0.1:9990 poetry run receptor --debug --node-id=node-a -d /tmp/node-a node --listen=ws://localhost:9998 --peer=ws://localhost:9999

Then I could observe that node-a was really using the proxy as the following was seen on the nginx access.log:

127.0.0.1 - - [19/May/2020:17:52:48 -0400] "GET ws://localhost:9999 HTTP/1.1" 502 157 "-" "Python/3.8 aiohttp/3.6.2"

The complete output of the controller node is the following:

$ poetry run receptor --debug --node-id=controller -d /tmp/controller node --listen=ws://localhost:9999
INFO 2020-05-19 16:10:31,348 controller entrypoints Running as Receptor node with ID: controller
INFO 2020-05-19 16:10:31,349 controller controller Serving on ws://localhost:9999

And the complete output of the node-a node is the following:

$ http_proxy=http://127.0.0.1:9990 poetry run receptor --debug --node-id=node-a -d /tmp/node-a node --listen=ws://localhost:9998 --peer=ws://localhost:9999
INFO 2020-05-19 17:54:19,148 node-a entrypoints Running as Receptor node with ID: node-a
INFO 2020-05-19 17:54:19,148 node-a controller Serving on ws://localhost:9998
INFO 2020-05-19 17:54:19,149 node-a controller Connecting to peer ws://localhost:9999
ERROR 2020-05-19 17:54:19,171 node-a ws ws.connect
Traceback (most recent call last):
  File "/home/elyezer/code/receptor/receptor/receptor/connection/ws.py", line 57, in connect
    async with aiohttp.ClientSession().ws_connect(
  File "/home/elyezer/.cache/pypoetry/virtualenvs/receptor-hhoD6yiq-py3.8/lib64/python3.8/site-packages/aiohttp/client.py", line 1012, in __aenter__
    self._resp = await self._coro
  File "/home/elyezer/.cache/pypoetry/virtualenvs/receptor-hhoD6yiq-py3.8/lib64/python3.8/site-packages/aiohttp/client.py", line 733, in _ws_connect
    raise WSServerHandshakeError(
aiohttp.client_exceptions.WSServerHandshakeError: 502, message='Invalid response status', url=URL('ws://localhost:9999')

The http_proxy can be either lowercase or uppercase and in both cases the same issue was encountered.

[elyezer]

After more investigation the configuration made for nginx provided on the previous comment happened to be a reverse proxy and we were looking for a forward proxy.

That said, we would be using squid to play the forward proxy role and we could not have it to work with ws. Which led us to have to use xss for the receptor connection.

To get receptor working with xss we had to first create a private CA which was done following the steps here https://github.com/ghjm/pki-stuff/blob/master/info.md. Thank you @ghjm very much for making that available and for all the help.

With a certificate generated and signed, I've started receptor by running:

$ poetry run receptor --debug --node-id=controller -d /tmp/controller --server-cert cert-signed.crt --server-key cert.key node --listen=wss://localhost:9999

And added the following line to the squid configuration:

acl SSL_ports port 9990-9999	# receptor testing

So the complete squid config was like the following:

acl localnet src 0.0.0.1-0.255.255.255	# RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8		# RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10		# RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16 	# RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12		# RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16		# RFC 1918 local private network (LAN)
acl localnet src fc00::/7       	# RFC 4193 local private network range
acl localnet src fe80::/10      	# RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl SSL_ports port 9990-9999	# receptor testing
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl CONNECT method CONNECT

http_access deny !Safe_ports

http_access deny CONNECT !SSL_ports

http_access allow localhost manager
http_access deny manager

http_access allow localnet
http_access allow localhost

http_access deny all

http_port 3128

coredump_dir /var/spool/squid

refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320

With squid running, then another receptor node was started by running:

$ https_proxy=http://localhost:3128 poetry run receptor --debug --node-id=node-a -d /tmp/node-a node --server-disable --peer=wss://localhost:9999

Here is the complete output for the controller node:

$ poetry run receptor --debug --node-id=controller -d /tmp/controller --server-cert cert-signed.crt --server-key cert.key node --listen=wss://localhost:9999
INFO 2020-05-20 13:02:22,247 controller entrypoints Running as Receptor node with ID: controller
DEBUG 2020-05-20 13:02:22,247 controller config Loading TLS Server Context
INFO 2020-05-20 13:02:22,261 controller controller Serving on wss://localhost:9999
DEBUG 2020-05-20 13:02:24,969 controller base TLSv1.3 connection with ('::1', 55374, 0, 0) using cipher TLS_AES_256_GCM_SHA384
                and certificate None.
DEBUG 2020-05-20 13:02:24,970 controller base waiting for HI
DEBUG 2020-05-20 13:02:24,971 controller base starting normal loop
DEBUG 2020-05-20 13:02:24,972 controller receptor spawning message_handler
DEBUG 2020-05-20 13:02:24,972 controller base Watching queue <receptor.connection.ws.WebSocket object at 0x7ff964c0bcd0>
DEBUG 2020-05-20 13:02:25,072 controller receptor Constructing routing table
DEBUG 2020-05-20 13:02:25,074 controller receptor    Routing updated. New table: [('controller', 'node-a', 1)]
DEBUG 2020-05-20 13:02:25,074 controller receptor Sending route advertisement aa7a99cb-94de-4022-8c8f-25f627fecaff seq 1
DEBUG 2020-05-20 13:02:25,074 controller receptor    Advertised connections: {'node-a': 1}
DEBUG 2020-05-20 13:02:25,075 controller receptor    Sent to node-a
DEBUG 2020-05-20 13:02:25,181 controller receptor Route advertisement 2fd8c4a0-322c-45f4-8b58-e1913b6961b2 seq 1 received From node-a via node-a
DEBUG 2020-05-20 13:02:25,181 controller receptor Constructing routing table
DEBUG 2020-05-20 13:02:25,182 controller receptor    Routing not changed. Existing table: [('controller', 'node-a', 1)]
DEBUG 2020-05-20 13:02:25,282 controller receptor Constructing routing table
DEBUG 2020-05-20 13:02:25,283 controller receptor    Routing not changed. Existing table: [('controller', 'node-a', 1)]
DEBUG 2020-05-20 13:02:25,283 controller receptor Sending route advertisement 5671da1f-c9d7-4009-99dd-3eabbd7f937e seq 2
DEBUG 2020-05-20 13:02:25,283 controller receptor    Advertised connections: {'node-a': 1}
DEBUG 2020-05-20 13:02:25,284 controller receptor    Sent to node-a
DEBUG 2020-05-20 13:02:25,388 controller receptor Route advertisement 6da40620-0739-494f-8f4a-c82d5e569913 seq 2 received From node-a via node-a
DEBUG 2020-05-20 13:02:25,388 controller receptor Constructing routing table
DEBUG 2020-05-20 13:02:25,390 controller receptor    Routing not changed. Existing table: [('controller', 'node-a', 1)]

And here is the complete output for the node-a node:

https_proxy=http://localhost:3128 poetry run receptor --debug --node-id=node-a -d /tmp/node-a node --server-disable --peer=wss://localhost:9999
INFO 2020-05-20 13:02:24,936 node-a entrypoints Running as Receptor node with ID: node-a
INFO 2020-05-20 13:02:24,936 node-a controller Connecting to peer wss://localhost:9999
DEBUG 2020-05-20 13:02:24,936 node-a config Loading TLS Client Context
DEBUG 2020-05-20 13:02:24,970 node-a base TLSv1.3 connection with ('::1', 3128, 0, 0) using cipher TLS_AES_256_GCM_SHA384
                and certificate {'subject': ((('countryName', 'US'),), (('localityName', 'Raleigh'),), (('organizationName', 'Receptor Inc'),), (('organizationalUnitName', 'QE'),), (('commonName', 'localhost'),)), 'issuer': ((('commonName', 'Easy-RSA CA'),),), 'version': 3, 'serialNumber': 'omitted', 'notBefore': 'May 20 16:45:23 2020 GMT', 'notAfter': 'Aug 23 16:45:23 2022 GMT', 'subjectAltName': (('DNS', 'localhost'),)}.
DEBUG 2020-05-20 13:02:24,970 node-a base waiting for HI
DEBUG 2020-05-20 13:02:24,972 node-a base starting normal loop
DEBUG 2020-05-20 13:02:24,973 node-a receptor spawning message_handler
DEBUG 2020-05-20 13:02:24,973 node-a base Watching queue <receptor.connection.ws.WebSocket object at 0x7fccaeb26d30>
DEBUG 2020-05-20 13:02:25,073 node-a receptor Constructing routing table
DEBUG 2020-05-20 13:02:25,075 node-a receptor    Routing updated. New table: [('controller', 'node-a', 1)]
DEBUG 2020-05-20 13:02:25,075 node-a receptor Sending route advertisement 2fd8c4a0-322c-45f4-8b58-e1913b6961b2 seq 1
DEBUG 2020-05-20 13:02:25,075 node-a receptor    Advertised connections: {'controller': 1}
DEBUG 2020-05-20 13:02:25,076 node-a receptor    Sent to controller
DEBUG 2020-05-20 13:02:25,180 node-a receptor Route advertisement aa7a99cb-94de-4022-8c8f-25f627fecaff seq 1 received From controller via controller
DEBUG 2020-05-20 13:02:25,180 node-a receptor Constructing routing table
DEBUG 2020-05-20 13:02:25,183 node-a receptor    Routing not changed. Existing table: [('controller', 'node-a', 1)]
DEBUG 2020-05-20 13:02:25,281 node-a receptor Constructing routing table
DEBUG 2020-05-20 13:02:25,282 node-a receptor    Routing not changed. Existing table: [('controller', 'node-a', 1)]
DEBUG 2020-05-20 13:02:25,283 node-a receptor Sending route advertisement 6da40620-0739-494f-8f4a-c82d5e569913 seq 2
DEBUG 2020-05-20 13:02:25,283 node-a receptor    Advertised connections: {'controller': 1}
DEBUG 2020-05-20 13:02:25,284 node-a receptor    Sent to controller
DEBUG 2020-05-20 13:02:25,388 node-a receptor Route advertisement 5671da1f-c9d7-4009-99dd-3eabbd7f937e seq 2 received From controller via controller
DEBUG 2020-05-20 13:02:25,389 node-a receptor Constructing routing table
DEBUG 2020-05-20 13:02:25,390 node-a receptor    Routing not changed. Existing table: [('controller', 'node-a', 1)]

With all that we can close this issue, again, after verifying that not only receptor was trying to use the proxy (as verified by ansible/receptor#191 (comment)) but also it really works end to end.