Timeout response when using a VPN (only few hosts)

[AlessandroZ]

Hello,

Sorry to open a new issue again, however, I had this problem various time without being able to debug it. Maybe it's a well known issue.

When I use a VPN, sometimes I have a timeout response on certain host.

> httpx -u scanme.sh -j -irh -debug -v

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

		projectdiscovery.io

[INF] Current httpx version v1.6.6 (latest)
[INF] Dumped HTTP request for https://scanme.sh

GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.55 Safari/537.36 Edg/96.0.1054.34
Accept-Charset: utf-8
Accept-Encoding: gzip

[INF] Dumped HTTP request for http://scanme.sh

GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36
Accept-Charset: utf-8
Accept-Encoding: gzip

[DBG] Failed 'http://scanme.sh': GET http://scanme.sh giving up after 1 attempts: Get "http://scanme.sh": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Whereas on the same host, using the same VPN, httprobe give me the good response.

> echo "scanme.sh" | httprobe 
https://scanme.sh
http://scanme.sh

When I connect to my vps and use httpx on the same domain, this time, it works.

> httpx -u scanme.sh -silent
https://scanme.sh

I'm sorry to not beeing able to give you more information, I can close this issue if you cannot reproduce it. It does not happen all the time but I faced this issue multiple times (it's not a recent problem). I checked it doing diff between httprobe and httpx results. I cannot explain why this happens on certain domains only and all the time httprobe does not miss it, so the way it does request should be different.

Have a nice day.

[dogancanbakir]

Using the -timeout flag, could you retry it with a different timeout value? The default is 10s

[AlessandroZ]

When I faced this problem, I tried changing these parameter but without success.
The best way should be to build manually to check exactly where the code fails but right now, I don't have enough time to do it. Sorry.

So same error as you could see:

> httpx -u scanme.sh -j -irh -debug -v -timeout 50

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

		projectdiscovery.io

[INF] Current httpx version v1.6.6 (latest)
[INF] Dumped HTTP request for https://scanme.sh

GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0
Accept-Charset: utf-8
Accept-Encoding: gzip

[INF] Dumped HTTP request for http://scanme.sh

GET / HTTP/1.1
Host: scanme.sh
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:70.0) Gecko/20100101 Firefox/70.0
Accept-Charset: utf-8
Accept-Encoding: gzip

[DBG] Failed 'http://scanme.sh': GET http://scanme.sh giving up after 1 attempts: Get "http://scanme.sh": [address=scanme.sh:80] context deadline exceeded

[dogancanbakir]

Closing this: I was able to repro the issue with the VPN gateways, which I suspect to be a bit problematic or sluggish. Therefore, it appears that we are left with no immediate solution.

[Techbrunch]

@AlessandroZ I probably ran into the same issue, are you on macOS by any chance ? In our testing this did not happen on linux.

[qu35t-code]

Hey all !

The issue seems to be related to DNS resolution through VPN. For me, the problem occurs with all domains (not few).

Here is a test that works via the VPN on the IP address of scanme.sh :

dig +short scanme.sh|httpx -v -sc

[WRN] UI Dashboard is disabled, Use -dashboard option to enable
https://128.199.158.128 [200]

The same test does not work via its domain name :

echo 'scanme.sh'|httpx -v -sc

[WRN] UI Dashboard is disabled, Use -dashboard option to enable
[DBG] Failed 'http://scanme.sh': GET http://scanme.sh giving up after 1 attempts: Get "http://scanme.sh": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

We can use the -pa flag and verbose mode, which confirms that the HTTP client failed to resolve the IP address of the domain scanme.sh :

echo 'scanme.sh'|httpx -v -sc -pa

[WRN] UI Dashboard is disabled, Use -dashboard option to enable
[DBG] Failed 'http://scanme.sh': GET http://scanme.sh giving up after 1 attempts: Get "http://scanme.sh": no ips provided in dialWrap; could not create dialwrap
https://scanme.sh [200] [128.199.158.128]

Using a proxy also works:

proxify -http-addr "127.0.0.1:9000"

[INF] HTTP Proxy Listening on 127.0.0.1:9000

echo 'scanme.sh'|httpx -v -sc -proxy http://127.0.0.1:9000

[WRN] UI Dashboard is disabled, Use -dashboard option to enable
https://scanme.sh [200]

Proxy logs :

{"timestamp":"2025-02-06T13:43:26+01:00","url":"https://scanme.sh:443","request":{"header":{"Connection":"close","User-Agent":"Go-http-client/1.1","host":"scanme.sh:443","method":"CONNECT","path":"","scheme":"https"},"raw":"CONNECT scanme.sh:443 HTTP/1.1\r\nHost: scanme.sh:443\r\nConnection: close\r\nUser-Agent: Go-http-client/1.1\r\n\r\n"},"response":{"raw":"HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n"}}
{"timestamp":"2025-02-06T13:43:27+01:00","url":"https://scanme.sh/","request":{"header":{"Accept-Charset":"utf-8","Accept-Encoding":"gzip","Connection":"close","User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36","host":"scanme.sh","method":"GET","path":"/","scheme":"https"},"raw":"GET / HTTP/1.1\r\nHost: scanme.sh\r\nAccept-Charset: utf-8\r\nAccept-Encoding: gzip\r\nConnection: close\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36\r\n\r\n"}}

[qu35t-code]

The bug could be caused by the retryabledns library: https://github.com/projectdiscovery/retryabledns

References :

• https://github.com/projectdiscovery/fastdialer/blob/f8e867309dce5eb5ebd6faacd7c6df82b43f4b8b/fastdialer/dialer_private.go#L167

• https://github.com/projectdiscovery/fastdialer/blob/f8e867309dce5eb5ebd6faacd7c6df82b43f4b8b/fastdialer/dialer_private.go#L115

• https://github.com/projectdiscovery/fastdialer/blob/main/fastdialer/dialer.go#L384

[qu35t-code]

I think the topic is legitimate to be reopened and a fix is possible. What do you think @dogancanbakir ?

[qu35t-code]

After some investigation with @lightdiscord, the issue might be related to IPv6 resolution. By analyzing the traffic in Wireshark, we can see that a call is made via IPv6, and it fails. We also see the same call using IPv4, which succeeds.

[dogancanbakir]

Let's take another look at it.