Timeout during connect (likely firewall problem)

[Celtech]

The problem

When attempting to issue a SSL certificate, we encounter this intermittent error:

could not obtain certificates:
	error: one or more domains had a problem:
[<REDACTED>.com] acme: error: 400 :: urn:ietf:params:acme:error:connection :: 148.<REDACTED>.14: Fetching http://<REDACTED>.com/.well-known/acme-challenge/8wbPIiG3xsLtzUED2CCSXiyOsNRDbdIUFzt9mfkyP3I: Timeout during connect (likely firewall problem) 

The problem is, it's not actually firewall related, 90% of the time we get this error but 10% of the time it just randomly works if you retry enough times. To me it seems like an HTTPS timeout issue as if its just on the edge, but that said, I am aware of the http-timeout and cert.timeout flag and I do have them set to 60 seconds but it still seems to time out way before 60 seconds which makes me think there's another flag or config option somewhere, just not sure where.

An important note:

I'm not using the lego CLI directly but rather importing lego as a package into my custom go app and using that to issue a certificate so the logging may look SLIGHTLY different than your normal logging but for the most part is the same.

Would be super helpful if anyone has any ideas! Thanks guys.

[ldez]

Hello,

urn:ietf:params:acme:error:connection is 100% of time a network issue:

• it can be related to requests from your system to an external endpoint
• or from an external request that tries to call your system

60s is already huge for a timeout. This timeout doesn't have an impact on external calls.

I guess you are using the HTTP-01 challenge, have you tried the TLS-ALPN-01 challenge?

[Celtech]

Idez,

Would you by chance have time to message over slack or discord about this issue? I wanna pick your brain further on this and doing it over these discussions may take awhile.

I'd be happy to post a resolution here once we find one.

[Celtech]

Edit: Looks like you were correct, it was infact a network issue, here's a detailed write-up in case anyone else encounters this weirdness:

In our case, our default gateway was borked on 2 of our docker swarm servers. But it was correct on the third server. Now these servers are clustered together as an ingress server.

Now before we can explain why this is borked we gotta understand our set up; We have 2 network cards, 1 for LAN and 1 for WAN. Now on the server that was working traffic would come in on wan and go back out on wan(the gateway was correct in this case). However on the borked servers the default gateway was set to LAN, so the traffic would come in on WAN, then try to go out on LAN and essentially get black holed inside our network.

So moral of the story? Make sure all your servers are configured the same and that your network is actually configured correctly!

Thanks for your help @ldez