[Bug]: Unable to connect to HostAccessPorts on container startup #2811
Comments
hathvi
changed the title
|
I probably should have just created a PR for this and we could discuss this further there. Let me know if you wish for me to do so and I'll find some time. |
Fixes testcontainers#2811 Previously ExposedHostPorts would start an SSHD container prior to starting the testcontainer and inject a PostReadies lifecycle hook into the testcontainer in order to set up remote port forwarding from the host to the SSHD container so the testcontainer can talk to the host via the SSHD container This would be an issue if the testcontainer depends on accessing the host port on startup ( e.g., a proxy server ) as the forwarding for the host access isn't set up until all the WiatFor strategies on the testcontainer have completed. The fix is to move the forwarding setup to the PostStarts hook on the testcontainer. Since remote forwarding doesn't establish a connect to the host port until a connection is made to the remote port, this should not be an issue even if the host isn't listening yet and ensures the remote port is available to the testcontainer immediately.
Fixes testcontainers#2811 Previously ExposedHostPorts would start an SSHD container prior to starting the testcontainer and inject a PostReadies lifecycle hook into the testcontainer in order to set up remote port forwarding from the host to the SSHD container so the testcontainer can talk to the host via the SSHD container This would be an issue if the testcontainer depends on accessing the host port on startup ( e.g., a proxy server ) as the forwarding for the host access isn't set up until all the WiatFor strategies on the testcontainer have completed. The fix is to move the forwarding setup to the PreCreates hook on the testcontainer. Since remote forwarding doesn't establish a connection to the host port until a connection is made to the remote port, this should not be an issue even if the host isn't listening yet and ensures the remote port is available to the testcontainer immediately.
|
I'm not sure if the approach here is correct, since the DNS entry for accessing the host should now work for all recent docker versions for example: package testcontainers_test
import (
"bytes"
"context"
"fmt"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"testing"
"github.com/docker/docker/api/types/container"
"github.com/stretchr/testify/require"
"github.com/testcontainers/testcontainers-go"
"github.com/testcontainers/testcontainers-go/wait"
)
const caddyFileContent = `
listen :80
reverse_proxy /api/* {
to {$API_SERVER}
health_uri /health
health_status 200
health_interval 10s
}
`
func TestCaddyfile(t *testing.T) {
ctx := context.Background()
apiServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello, World!")
}))
t.Cleanup(apiServer.Close)
u, err := url.Parse(apiServer.URL)
require.NoError(t, err)
_, port, err := net.SplitHostPort(u.Host)
require.NoError(t, err)
u.Host = "host.docker.internal:" + port
caddyContainer, err := testcontainers.GenericContainer(ctx, testcontainers.GenericContainerRequest{
ContainerRequest: testcontainers.ContainerRequest{
Image: "caddy:2.8.4",
ExposedPorts: []string{"80/tcp"},
WaitingFor: wait.ForLog("server running"),
Env: map[string]string{
"API_SERVER": u.String(),
},
Files: []testcontainers.ContainerFile{
{
Reader: bytes.NewReader([]byte(caddyFileContent)),
ContainerFilePath: "/etc/caddy/Caddyfile",
},
},
HostConfigModifier: func(hc *container.HostConfig) {
hc.ExtraHosts = append(hc.ExtraHosts, "host.docker.internal:host-gateway")
},
},
Started: true,
})
testcontainers.CleanupContainer(t, caddyContainer)
require.NoError(t, err)
caddyURL, err := caddyContainer.PortEndpoint(ctx, "80/tcp", "http")
require.NoError(t, err)
resp, err := http.Get(caddyURL + "/api/test")
require.NoError(t, err)
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
require.NoError(t, err)
require.Equal(t, http.StatusOK, resp.StatusCode)
require.Equal(t, "Hello, World!\n", string(body))
lr, err := caddyContainer.Logs(ctx)
require.NoError(t, err)
lb, err := io.ReadAll(lr)
require.NoError(t, err)
fmt.Printf("== Caddy Logs ==\n%s================\n\n", string(lb))
} |
|
Thanks @stevenh! I like this approach better for my use case but I'll keep this issue and PR open and let the project maintainers decide if it's worth pulling in? Given your solution, I'm not sure what the use case is for Thank you for your insight! |
|
One of the benefits of the |
I'm not sure I understand, |
As one of the community maintainers, happy to consider your PR, I'm just trying to understand the use case, and if its really needed anymore? I'm wondering if |
|
I just came across this issue yesterday as I was exploring a similar problem with The DNS solution works great and is certainly simpler, so thanks for that @stevenh ! My $.02 -- IMO, if the support for If indeed there is no justification for continuing to support |
|
Thanks for your thoughts @jeremyg484, can anyone think of a situation which |
|
Are you two by chance on Mac or Windows? I think Windows, under Docker Desktop, routes traffic to the host a bit differently then Mac or Linux. I use Linux as my primary OS and am not super familiar with Windows. The test that @stevenh provided does not work for me, which I'll describe why below. I'd preface that my original problem with testcontainers was that it was non obvious how to access the host using testcontainers. I however was also not aware of the docker extra hosts config and my searching around testcontainers lead me to Docker by default uses a bridge network. It creates a new interface on the host named When you create a new
My guess would be that you guys are likely on Windows and/or running Docker Desktop which likely does something similar to what I would say I would also say that if |
|
I don't contribute as much to open source as I should mainly due to anxiety socializing with others, but I'd be down to contributing the suggested changes if that seems like a good idea to you guys. |
|
Actually, given this a tiny bit more thought, I'm not sure my additional suggested changes to |
|
Thanks for all the extra info. I wonder what the expected behaviour should be. Could you confirm what the name resolves to? |
|
Oh to confirm my test host was Windows under WSL. So would be interested to understand that OSs work and what doesn't |
|
Tested on Mac and it also works fine. |
Thats still a legitimate problem to solve. |
If we're referring to DNS using the extra hosts set to If I listen on If I listen on
To clarify, I was only referring to my suggested changes on adding the ability to specify the local address to connect to in It should only be a problem if the server you're trying to talk to is bound to an address on the host that is not routable from the container. This is the case for 127.0.0.1 as the container also creates an loopback interface with an addr of 127.0.0.1, so talking to 127.0.0.1 would just send to the container loopback interface and not the host. For example I can connect to my tailscale interface from a container just fine as the container doesn't have a route for this address and sends the request out the default gateway to docker0 on the host, and the host has a route to the tailscale address so sends it out that interface. I think there's a legitimate helpful use case for
I'm not sure how/why this is working on windows and macos unless binding a port on 127.0.0.1 for some results in it listening on all interfaces. On your Mac are you also using Docker Desktop? I still speculate that Docker Desktop is doing something to expose the host to the VM that results in ports bound on 127.0.0.1 working from the container. |
|
What does |
|
To add some further examples of usage to the discussion - I have not tested on Windows, but my specific usage of I am setting I am setting a custom according to the docs for I just checked, and if I specify the loopback address |
|
That's what I see too, Linux seems to be the odd one out. |
|
I wouldn't call Linux the odd one out, it's working as it should, I think Mac, Windows or something else is just configured slightly different within these test environments and those are the odd ones out. Docker is Linux native, and Docker for me is working just as I would expect given the defaults the Docker documentation define and how networking in general works both inside and outside containers. Can you run a few commands @stevenh?
I can set up a Windows VM later tonight ~8PM PT w/ Docker Desktop and WSL, just so I can explore more mainly out of curiosity now. I think given the differences we see just in this thread gives |
docker run --rm -it --add-host host.docker.internal:host-gateway alpine cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.65.254 host.docker.internal
172.17.0.2 32db41e7dd87
docker inspect 020989b419e6 -f '{{json .NetworkSettings.Networks}}'
{"bridge":{"IPAMConfig":null,"Links":null,"Aliases":null,"MacAddress":"02:42:ac:11:00:03","DriverOpts":null,"NetworkID":"f4e593f77e57e8d759b34578d47ca89583c3cae50f4ac18b5cf9d447a97cebb2","EndpointID":"0dcb292a563a3041263672990cce0c49efcb28b60f3eca4e6d3614d82ac7d31b","Gateway":"172.17.0.1","IPAddress":"172.17.0.3","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"DNSNames":null}}
netstat -na |grep 45757 |grep LIST
tcp 30 0 127.0.0.1:45757 0.0.0.0:* LISTEN
docker version
Client:
Version: 27.2.0
API version: 1.47
Go version: go1.21.13
Git commit: 3ab4256
Built: Tue Aug 27 14:14:20 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Desktop ()
Engine:
Version: 27.2.0
API version: 1.47 (minimum version 1.24)
Go version: go1.21.13
Git commit: 3ab5c7d
Built: Tue Aug 27 14:15:15 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.20
GitCommit: 8fc6bcff51318944179630522a095cc9dbf9f353
runc:
Version: 1.1.13
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0
We shouldn't assume that because docker is native on Linux that the intent wasn't that seen on Windows or Mac with regards being able to connect to loopback address. It could be a bug with either, as we should be able to rely on consistent behaviour across all platforms. |
|
I was not able to get WSL2 running within VirtualBox last night, but I got things set up on a laptop this AM and can confirm your test works for me on Windows as well. Based on the outputs you provided in your last post this should technically not be allowed unless something in the middle is doing something with the network packets such as NAT or proxying to the host. It looks like WSL2 uses NAT by default, but doesn't allow this behavior itself. server.gopackage main
import (
"fmt"
"net"
"net/http"
"net/http/httptest"
"os"
"os/signal"
"syscall"
)
func main() {
l, err := net.Listen("tcp", "127.0.0.1:51403")
if err != nil {
panic(err)
}
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
w.Write([]byte("Hello, world!"))
}))
s.Listener.Close()
s.Listener = l
s.Start()
fmt.Printf("Listening: %s\n", s.Listener.Addr())
sc := make(chan os.Signal, 1)
signal.Notify(sc, syscall.SIGTERM)
<-sc
}The WSL2 documentation even makes explicit mention of this https://learn.microsoft.com/en-us/windows/wsl/networking#connecting-via-remote-ip-addresses
I tried digging more into how Docker is running under WSL2 but at first glance doesn't seem as straight forward as I was hoping. I'll try digging more into this later tonight. |
Testcontainers version
0.33.0
Using the latest Testcontainers version?
Yes
Host OS
Linux
Host arch
x86_64
Go version
1.23.1
Docker version
Client: Docker Engine - Community
Version: 27.3.1
API version: 1.47
Go version: go1.22.7
Git commit: ce12230
Built: Fri Sep 20 11:41:00 2024
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 27.3.1
API version: 1.47 (minimum version 1.24)
Go version: go1.22.7
Git commit: 41ca978
Built: Fri Sep 20 11:41:00 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.22
GitCommit: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc:
Version: 1.1.14
GitCommit: v1.1.14-0-g2c9f560
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Docker info
Client: Docker Engine - Community
Version: 27.3.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.17.1
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.29.7
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 14
Running: 9
Paused: 0
Stopped: 5
Images: 34
Server Version: 27.3.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
runc version: v1.1.14-0-g2c9f560
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-40-generic
Operating System: Ubuntu 22.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 62.52GiB
Name: jhome
ID: 43fdd48e-011e-40da-aff1-b76bc378d203
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
What happened?
I'm attempting to utilize testcontainers-go to test my Caddy configuration as a gateway to my API server but I'm running into problems with how testcontainers-go exposes host ports and I believe this issue to be a bug.
Setup
In my tests, I've set up a
httptest.Serverto act as my API server, listening on a random port on the host. I then set up Caddy in a testcontainer and expose the API server port to the container viaHostAccessPorts. My Caddy configuration defines the API server with a health check which Caddy checks on startup.caddyfile_test.go
Test Output
== Caddy Logs == {"level":"info","ts":1727952070.1965187,"msg":"using config from file","file":"/etc/caddy/Caddyfile"} {"level":"info","ts":1727952070.1969736,"msg":"adapted config to JSON","adapter":"caddyfile"} {"level":"warn","ts":1727952070.1969776,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":1} {"level":"info","ts":1727952070.1972885,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]} {"level":"warn","ts":1727952070.1973321,"logger":"http.auto_https","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv1","http_port":80} {"level":"info","ts":1727952070.1973393,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443} {"level":"info","ts":1727952070.1973433,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"} {"level":"info","ts":1727952070.1973994,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000736a80"} {"level":"info","ts":1727952070.1974878,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"} {"level":"info","ts":1727952070.197532,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."} {"level":"info","ts":1727952070.1975832,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]} +{"level":"info","ts":1727952070.1976013,"logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]} {"level":"info","ts":1727952070.1976032,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["listen"]} -{"level":"info","ts":1727952070.1976056,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"host.testcontainers.internal:43017","error":"Get \"http://host.testcontainers.internal:43017/health\": dial tcp 172.17.0.3:43017: connect: connection refused"} -{"level":"info","ts":1727952070.1976073,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"HTTP request failed","host":"host.testcontainers.internal:43017","error":"Get \"http://host.testcontainers.internal:43017/health\": dial tcp 172.17.0.3:43017: connect: connection refused"} {"level":"info","ts":1727952070.1978004,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"} {"level":"info","ts":1727952070.1978037,"msg":"serving initial configuration"} {"level":"info","ts":1727952070.197835,"logger":"tls.obtain","msg":"acquiring lock","identifier":"listen"} {"level":"info","ts":1727952070.1985145,"logger":"tls.obtain","msg":"lock acquired","identifier":"listen"} {"level":"info","ts":1727952070.1985347,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"listen"} {"level":"info","ts":1727952070.1985307,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"} {"level":"info","ts":1727952070.1986136,"logger":"tls","msg":"finished cleaning storage units"} -{"level":"error","ts":1727952070.3384068,"logger":"http.log.error","msg":"no upstreams available","request":{"remote_ip":"172.17.0.1","remote_port":"54434","client_ip":"172.17.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost:33500","uri":"/api/test","headers":{"User-Agent":["Go-http-client/1.1"],"Accept-Encoding":["gzip"]}},"duration":0.000040091,"status":503,"err_id":"qe8hu1acn","err_trace":"reverseproxy.(*Handler).proxyLoopIteration (reverseproxy.go:486)"} ================ --- FAIL: TestCaddyfile (1.10s) caddyfile_test.go:76: Error Trace: /home/justin/workspace/test/caddyfile_test.go:76 Error: Not equal: expected: 200 actual : 503 Test: TestCaddyfile caddyfile_test.go:77: Error Trace: /home/justin/workspace/test/caddyfile_test.go:77 Error: Not equal: expected: "Hello, World!\n" actual : "" Diff: --- Expected +++ Actual @@ -1,2 +1 @@ -Hello, World! Test: TestCaddyfile FAIL FAIL github.com/hathvi/test 1.189s FAILProblem
My problem with this set up is that Caddy logs a "connection refused" error for the health check even though the testcontainer is ready. I attempt to make a request to the Caddy server after startup but receive an HTTP 502 Bad Gateway error as the API server wasn't initially reachable even though it's running and accepting connections on the host. Caddy will continue to return an HTTP 502 until the next health check.
My Analysis
I can see that
HostAccessPortsutilizes a separate container running an SSH server then sets up aPostReadieslifecycle hook on the Caddy container in order to then set up the forwarding in the SSH container. It appears to do the forwarding by firing off a go routine that connects to the SSH container with remote port forwarding, listening to theHostAccessPortsports on the SSH container and tunneling this to the host.PostReadiesseems like a lot to late to setup the forwarding. I'm utilizingHostAccessPortsso I can talk to a server on the host from my testcontainer, so in order for my test container to be ready I would expect to be able to talk to that server before I do any of my testing. Logically I would assume I should be able to have a wait strategy that depends on that connection being made.Test fix
I created a fork and updated
exposeHostPortsto setup a lifecycle hook onPreCreatesinstead ofPostReadies. This ensures the host port is accessible via the SSH container to the testcontainer from all lifecycle hooks and container command.In theory this shouldn't break anything even if someone sets up the listener for the host port in a later lifecycle hook as connections back on the host port are only established once you try connecting to the remote port.
testcontainers-go.patch
Relevant log output
No response
Additional information
No response
The text was updated successfully, but these errors were encountered: