Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: create container: port not found: creating reaper failed #2798

Open
mukulmantosh opened this issue Sep 25, 2024 · 9 comments
Open

[Bug]: create container: port not found: creating reaper failed #2798

mukulmantosh opened this issue Sep 25, 2024 · 9 comments
Labels
bug An issue with the library

Comments

@mukulmantosh
Copy link

Testcontainers version

0.33.0

Using the latest Testcontainers version?

Yes

Host OS

Windows

Host arch

x86

Go version

1.23

Docker version

Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Aug 27 14:17:17 2024
 OS/Arch:           windows/amd64
 Context:           desktop-linux

Server: Docker Desktop 4.34.2 (167172)
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:15 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Docker info

Client:
 Version:    27.2.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     C:\Program Files\Docker\cli-plugins\docker-buildx.exe
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-compose.exe
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     C:\Program Files\Docker\cli-plugins\docker-debug.exe
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     C:\Program Files\Docker\cli-plugins\docker-desktop.exe
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     C:\Program Files\Docker\cli-plugins\docker-dev.exe
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     C:\Program Files\Docker\cli-plugins\docker-extension.exe
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     C:\Program Files\Docker\cli-plugins\docker-feedback.exe
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-init.exe
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-sbom.exe
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     C:\Program Files\Docker\cli-plugins\docker-scout.exe

Server:
 Containers: 35
  Running: 19
  Paused: 0
  Stopped: 16
 Images: 19
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: nvidia runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 31.24GiB
 Name: docker-desktop
 ID: d257b354-0307-46dc-98db-5e5637a2026d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=npipe://\\.\pipe\docker_cli
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

What happened?

I attempted to run tests that involve both Postgres and NATS. While they often run smoothly on their own, there are times when they fail unexpectedly.

I am not using the GenericContainer, instead using the official supported ones.

Example: Postgres

ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()

pgContainer, err = postgres.Run(ctx, containerImage,
	postgres.WithDatabase(dbName),
	postgres.WithUsername(dbUsername),
	postgres.WithPassword(dbPassword),
	testcontainers.WithWaitStrategy(
		wait.ForLog("database system is ready to accept connections").
			WithOccurrence(2).WithStartupTimeout(10*time.Second)),
)
if err != nil {
	log.Fatalf("failed to start PostgreSQL container: %s", err)
}

// Set environment variables
dbHost, _ = pgContainer.Host(ctx)
mappedPort, _ := pgContainer.MappedPort(ctx, "5432/tcp")
dbPort = mappedPort.Port()
_ = os.Setenv("DB_USERNAME", dbUsername)
_ = os.Setenv("DB_PASSWORD", dbPassword)
_ = os.Setenv("DB_NAME", dbName)
_ = os.Setenv("DB_PORT", dbPort)
_ = os.Setenv("DB_HOST", dbHost)

NATS

natsContainer, err := nats.Run(ctx, "nats:2.10.20", testcontainers.WithHostPortAccess(4222))

if err != nil {
	t.Logf("failed to start NATS container: %s", err)
	return
}

Relevant log output

failed to start NATS container: create container: port not found: creating reaper failed

Additional information

No response

@mukulmantosh mukulmantosh added the bug An issue with the library label Sep 25, 2024
@mukulmantosh
Copy link
Author

go.mod

github.com/testcontainers/testcontainers-go v0.33.0
github.com/testcontainers/testcontainers-go/modules/nats v0.33.0
github.com/testcontainers/testcontainers-go/modules/postgres v0.32.0

@stevenh
Copy link
Collaborator

stevenh commented Oct 7, 2024

There are know race conditions in the reaper which should be fixed once we've merged #2728 and updated to use the new reaper image container the work which was merged today testcontainers/moby-ryuk#141

@erikmansson
Copy link

@mukulmantosh did you find any workaround for this issue?

@stevenh
Copy link
Collaborator

stevenh commented Oct 14, 2024

If you use the branch from #2728 does it fix your issue?

@erikmansson
Copy link

On the merged commit, we now instead get

        	Error:      	Received unexpected error:
        	            	generic container: create container: reaper: from container "1aabe62f": wait for reaper 1aabe62f: external check: check target: retries: 571 address: 172.19.0.1:58322: get state: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.46/containers/1aabe62f3a5988e7bf88dbae63aa756f9700f3c6697eb41fa6b5f56933bb080c/json": context deadline exceeded

@stevenh
Copy link
Collaborator

stevenh commented Oct 21, 2024

Can you confirm you're using v0.34.0 and the exact test case you're using?

@stevenh
Copy link
Collaborator

stevenh commented Oct 21, 2024

I've tried with the following and all works as expected, so suspecting it could be something setup releated

package testcontainers_test

import (
	"context"
	"testing"
	"time"

	"github.com/stretchr/testify/require"
	"github.com/testcontainers/testcontainers-go"
	"github.com/testcontainers/testcontainers-go/modules/postgres"
	"github.com/testcontainers/testcontainers-go/wait"
)

func TestReaper(t *testing.T) {
	ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
	defer cancel()

	dbName, dbUsername, dbPassword := "test_db", "test_user", "test_password"

	pgContainer, err := postgres.Run(ctx, "docker.io/postgres:16-alpine",
		postgres.WithDatabase(dbName),
		postgres.WithUsername(dbUsername),
		postgres.WithPassword(dbPassword),
		testcontainers.WithWaitStrategy(
			wait.ForLog("database system is ready to accept connections").
				WithOccurrence(2).WithStartupTimeout(10*time.Second),
		),
	)
	testcontainers.CleanupContainer(t, pgContainer)
	require.NoError(t, err)

	dbHost, err := pgContainer.Host(ctx)
	require.NoError(t, err)
	require.NotEmpty(t, dbHost)

	mappedPort, err := pgContainer.MappedPort(ctx, "5432/tcp")
	require.NoError(t, err)
	require.NotEmpty(t, mappedPort)
}

go.mod:

module github.com/testcontainers/testcontainers-go/test

go 1.23.2

require (
        github.com/stretchr/testify v1.9.0
        github.com/testcontainers/testcontainers-go v0.34.0
        github.com/testcontainers/testcontainers-go/modules/postgres v0.34.0
)

require (
        dario.cat/mergo v1.0.0 // indirect
        github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
        github.com/Microsoft/go-winio v0.6.2 // indirect
        github.com/cenkalti/backoff/v4 v4.2.1 // indirect
        github.com/containerd/containerd v1.7.18 // indirect
        github.com/containerd/log v0.1.0 // indirect
        github.com/containerd/platforms v0.2.1 // indirect
        github.com/cpuguy83/dockercfg v0.3.2 // indirect
        github.com/davecgh/go-spew v1.1.1 // indirect
        github.com/distribution/reference v0.6.0 // indirect
        github.com/docker/docker v27.1.1+incompatible // indirect
        github.com/docker/go-connections v0.5.0 // indirect
        github.com/docker/go-units v0.5.0 // indirect
        github.com/felixge/httpsnoop v1.0.4 // indirect
        github.com/go-logr/logr v1.4.1 // indirect
        github.com/go-logr/stdr v1.2.2 // indirect
        github.com/go-ole/go-ole v1.2.6 // indirect
        github.com/gogo/protobuf v1.3.2 // indirect
        github.com/google/uuid v1.6.0 // indirect
        github.com/klauspost/compress v1.17.4 // indirect
        github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
        github.com/magiconair/properties v1.8.7 // indirect
        github.com/moby/docker-image-spec v1.3.1 // indirect
        github.com/moby/patternmatcher v0.6.0 // indirect
        github.com/moby/sys/sequential v0.5.0 // indirect
        github.com/moby/sys/user v0.1.0 // indirect
        github.com/moby/term v0.5.0 // indirect
        github.com/morikuni/aec v1.0.0 // indirect
        github.com/opencontainers/go-digest v1.0.0 // indirect
        github.com/opencontainers/image-spec v1.1.0 // indirect
        github.com/pkg/errors v0.9.1 // indirect
        github.com/pmezard/go-difflib v1.0.0 // indirect
        github.com/power-devops/perfstat v0.0.0-20210106213030-5aafc221ea8c // indirect
        github.com/shirou/gopsutil/v3 v3.23.12 // indirect
        github.com/shoenig/go-m1cpu v0.1.6 // indirect
        github.com/sirupsen/logrus v1.9.3 // indirect
        github.com/tklauser/go-sysconf v0.3.12 // indirect
        github.com/tklauser/numcpus v0.6.1 // indirect
        github.com/yusufpapurcu/wmi v1.2.3 // indirect
        go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
        go.opentelemetry.io/otel v1.24.0 // indirect
        go.opentelemetry.io/otel/metric v1.24.0 // indirect
        go.opentelemetry.io/otel/trace v1.24.0 // indirect
        golang.org/x/crypto v0.24.0 // indirect
        golang.org/x/sys v0.21.0 // indirect
        gopkg.in/yaml.v3 v3.0.1 // indirect
)

@erikmansson
Copy link

We later tried v0.34.0 as well - no difference. However, we did figure out that it has something to do with running the test inside a Docker container with /var/run/docker.sock mounted. Guess that's a separate issue.

@weihong-xu-hpe
Copy link

We saw recently the same on github actions more and more common for this issues.
Running at github codespace those issues are gone.
v0.33.0 and v0.34.0 get no difference, always some random testcontainer connection failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug An issue with the library
Projects
None yet
Development

No branches or pull requests

4 participants