Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(HTTP code 409) container stopped/paused - container ... is not running #881

Closed
mejenborg opened this issue Dec 6, 2024 · 2 comments
Closed

(HTTP code 409) container stopped/paused - container ... is not running #881

mejenborg opened this issue Dec 6, 2024 · 2 comments

Comments

@mejenborg
Copy link

Expected Behaviour
That a database container was successfully launched

Actual Behaviour
It fails launching

Testcontainer Logs

  testcontainers [DEBUG] Checking container runtime strategy "TestcontainersHostStrategy"... +0ms
  testcontainers [DEBUG] Container runtime strategy "TestcontainersHostStrategy" is not applicable +1ms
  testcontainers [DEBUG] Checking container runtime strategy "ConfigurationStrategy"... +0ms
  testcontainers [DEBUG] Container runtime strategy "ConfigurationStrategy" is not applicable +0ms
  testcontainers [DEBUG] Checking container runtime strategy "UnixSocketStrategy"... +0ms
  testcontainers [TRACE] Fetching Docker info... +0ms
  testcontainers [TRACE] Fetching remote container runtime socket path... +17ms
  testcontainers [TRACE] Resolving host... +0ms
  testcontainers [TRACE] Fetching Compose info... +0ms
  testcontainers [TRACE] Looking up host IPs... +810ms
  testcontainers [TRACE] Initialising clients... +18ms
  testcontainers [TRACE] Container runtime info:
  testcontainers {
  testcontainers   "node": {
  testcontainers     "version": "v22.11.0",
  testcontainers     "architecture": "arm64",
  testcontainers     "platform": "darwin"
  testcontainers   },
  testcontainers   "containerRuntime": {
  testcontainers     "host": "localhost",
  testcontainers     "hostIps": [
  testcontainers       {
  testcontainers         "address": "::1",
  testcontainers         "family": 6
  testcontainers       },
  testcontainers       {
  testcontainers         "address": "127.0.0.1",
  testcontainers         "family": 4
  testcontainers       }
  testcontainers     ],
  testcontainers     "remoteSocketPath": "/var/run/docker.sock",
  testcontainers     "indexServerAddress": "https://index.docker.io/v1/",
  testcontainers     "serverVersion": "27.3.1",
  testcontainers     "operatingSystem": "Docker Desktop",
  testcontainers     "operatingSystemType": "linux",
  testcontainers     "architecture": "aarch64",
  testcontainers     "cpus": 10,
  testcontainers     "memory": 8217968640,
  testcontainers     "runtimes": [
  testcontainers       "io.containerd.runc.v2",
  testcontainers       "runc"
  testcontainers     ],
  testcontainers     "labels": [
  testcontainers       "com.docker.desktop.address=unix:///Users/dkKasSoe/Library/Containers/com.docker.docker/Data/docker-cli.sock"
  testcontainers     ]
  testcontainers   },
  testcontainers   "compose": {
  testcontainers     "version": "2.30.3-desktop.1",
  testcontainers     "compatability": "v2"
  testcontainers   }
  testcontainers } +1ms
  testcontainers [DEBUG] Container runtime strategy "UnixSocketStrategy" works +1ms
  testcontainers [DEBUG] Checking if image exists "mysql:8.0.31"... +1ms
  testcontainers [DEBUG] Checked if image exists "mysql:8.0.31" +9ms
  testcontainers [DEBUG] Image "mysql:8.0.31" already exists +1ms
  testcontainers [DEBUG] Acquiring lock file "/var/folders/dt/y51xshd153zc7rm15rdhyqdc0000gp/T/testcontainers-node.lock"... +40ms
  testcontainers [DEBUG] Acquired lock file "/var/folders/dt/y51xshd153zc7rm15rdhyqdc0000gp/T/testcontainers-node.lock" +4ms
  testcontainers [DEBUG] Listing containers... +0ms
  testcontainers [DEBUG] Listed containers +8ms
  testcontainers [DEBUG] Creating new Reaper for session "7f5af60dfd7c" with socket path "/var/run/docker.sock"... +1ms
  testcontainers [DEBUG] Checking if image exists "testcontainers/ryuk:0.5.1"... +0ms
  testcontainers [DEBUG] Checked if image exists "testcontainers/ryuk:0.5.1" +7ms
  testcontainers [DEBUG] Image "testcontainers/ryuk:0.5.1" already exists +0ms
  testcontainers [DEBUG] Creating container for image "testcontainers/ryuk:0.5.1"... +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Created container for image "testcontainers/ryuk:0.5.1" +31ms
  testcontainers [INFO] [d25b93c75e8c] Starting container for image "testcontainers/ryuk:0.5.1"... +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Starting container... +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Started container +92ms
  testcontainers [INFO] [d25b93c75e8c] Started container for image "testcontainers/ryuk:0.5.1" +1ms
  testcontainers [DEBUG] [d25b93c75e8c] Inspecting container... +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Inspected container +3ms
  testcontainers [DEBUG] [d25b93c75e8c] Fetching container logs... +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Demuxing stream... +3ms
  testcontainers [DEBUG] [d25b93c75e8c] Demuxed stream +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Fetched container logs +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Waiting for container to be ready... +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Waiting for log message "/.+ Started!/"... +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Fetching container logs... +0ms
  testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Pinging Docker... +0ms
  testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Docker daemon is available! +0ms
  testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Starting on port 8080... +0ms
  testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Started! +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Demuxing stream... +3ms
  testcontainers [DEBUG] [d25b93c75e8c] Demuxed stream +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Fetched container logs +1ms
  testcontainers [DEBUG] [d25b93c75e8c] Log wait strategy complete +2ms
  testcontainers [INFO] [d25b93c75e8c] Container is ready +1ms
  testcontainers [DEBUG] [d25b93c75e8c] Connecting to Reaper (attempt 1) on "localhost:55012"... +0ms
  testcontainers [DEBUG] [d25b93c75e8c] Connected to Reaper +2ms
  testcontainers [DEBUG] Releasing lock file "/var/folders/dt/y51xshd153zc7rm15rdhyqdc0000gp/T/testcontainers-node.lock"... +0ms
  testcontainers [DEBUG] Released lock file "/var/folders/dt/y51xshd153zc7rm15rdhyqdc0000gp/T/testcontainers-node.lock" +1ms
  testcontainers [DEBUG] Creating container for image "mysql:8.0.31"... +0ms
  testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 New client connected: 172.17.0.1:57630 +12ms
  testcontainers:containers [d25b93c75e8c] 2024/12/06 23:08:33 Adding {"label":{"org.testcontainers.session-id=7f5af60dfd7c":true}} +2ms
  testcontainers [DEBUG] [39f675b64955] Created container for image "mysql:8.0.31" +23ms
  testcontainers [INFO] [39f675b64955] Starting container for image "mysql:8.0.31"... +0ms
  testcontainers [DEBUG] [39f675b64955] Starting container... +1ms
  testcontainers [DEBUG] [39f675b64955] Started container +99ms
  testcontainers [INFO] [39f675b64955] Started container for image "mysql:8.0.31" +0ms
  testcontainers [DEBUG] [39f675b64955] Inspecting container... +0ms
  testcontainers [DEBUG] [39f675b64955] Inspected container +2ms
  testcontainers [DEBUG] [39f675b64955] Fetching container logs... +0ms
  testcontainers [DEBUG] [39f675b64955] Demuxing stream... +5ms
  testcontainers [DEBUG] [39f675b64955] Demuxed stream +0ms
  testcontainers [DEBUG] [39f675b64955] Fetched container logs +0ms
  testcontainers [DEBUG] [39f675b64955] Waiting for container to be ready... +0ms
  testcontainers [DEBUG] [39f675b64955] Waiting for host port 55013... +1ms
  testcontainers [DEBUG] [39f675b64955] Waiting for internal port 3306... +0ms
  testcontainers:containers [39f675b64955] 2024-12-06 23:08:33+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.31-1.el8 started. +127ms
  testcontainers [DEBUG] [39f675b64955] Host port 55013 ready +3ms
  testcontainers [DEBUG] [39f675b64955] Host port wait strategy complete +0ms
  testcontainers [TRACE] [39f675b64955] Port check result exit code 127: /bin/sh: nc: command not found +41ms
  testcontainers [TRACE] [39f675b64955] Port check result exit code 1: /bin/bash: connect: Connection refused
  testcontainers /bin/bash: /dev/tcp/localhost/3306: Connection refused +0ms
  testcontainers:containers [39f675b64955] 2024-12-06 23:08:33+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' +72ms
  testcontainers:containers [39f675b64955] 2024-12-06 23:08:33+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.31-1.el8 started. +4ms
  testcontainers:containers [39f675b64955] 2024-12-06 23:08:33+00:00 [Note] [Entrypoint]: Initializing database files +69ms
  testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.806686Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead. +4ms
  testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.806729Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.31) initializing of server in progress as process 100 +0ms
  testcontainers:containers [39f675b64955] mysqld: Error writing file '/var/lib/mysql/auto.cnf' (OS errno 28 - No space left on device) +1ms
  testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.809446Z 0 [ERROR] [MY-010076] [Server] Initialization of the server's UUID failed because it could not be read from the auto.cnf file. If this is a new server, the initialization failed because it was not possible to generate a new UUID. +0ms
  testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.809453Z 0 [ERROR] [MY-013236] [Server] The designated data directory /var/lib/mysql/ is unusable. You can remove all files that the server added to it. +1ms
  testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.809456Z 0 [ERROR] [MY-010119] [Server] Aborting +0ms
  testcontainers:containers [39f675b64955] 2024-12-06T23:08:33.809556Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.31)  MySQL Community Server - GPL. +0ms
  testcontainers [ERROR] [39f675b64955] The HostPortWaitStrategy will not work on a distroless image, use an alternate wait strategy +118ms
  testcontainers [ERROR] [39f675b64955] Failed to exec container with command "/bin/bash -c </dev/tcp/localhost/3306": Error: (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running :  +106ms
  testcontainers [ERROR] [39f675b64955] Container failed to be ready: Error: (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running  +0ms
  testcontainers [DEBUG] [39f675b64955] Stopping container... +1ms
  testcontainers [ERROR] [39f675b64955] Failed to exec container with command "/bin/sh -c cat /proc/net/tcp* | awk '{print $2}' | grep -i :0cea": Error: (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running :  +1ms
  testcontainers [ERROR] [39f675b64955] Failed to exec container with command "/bin/sh -c nc -vz -w 1 localhost 3306": Error: (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running :  +2ms
  testcontainers [DEBUG] [39f675b64955] Container already stopped +0ms
  testcontainers [DEBUG] [39f675b64955] Removing container... +1ms
  testcontainers [DEBUG] [39f675b64955] Removed container +5ms

Steps to Reproduce
I've reproduced the error in this repository https://github.com/mejenborg/testcontainers-node-fails
It's a clean NodeJS project, with typeorm, @nestjs/typeorm, mysql2, and @testcontainers/mysql packages installed.
When executing $ npm run test, all tests fails with (HTTP code 409) container stopped/paused - container 39f675b64955b7b5176795361fe5bf42a9a2cffe4a362d1f99ad78d49b02e78c is not running

Environment Information

  • Operating System: Mac OS Sequoia 15.1.1
  • Docker Version: 27.3.1
  • Node version: 22.11.0
  • NestJS version: 10.4.13
  • Jest version: 29.7.0
  • Testcontainers version: 10.16.0
@gustavotrapp
Copy link

Also have this problem, with a similar setup. Any updates

@mejenborg
Copy link
Author

I was so determined that these two lines were the root cause:

testcontainers [TRACE] [39f675b64955] Port check result exit code 127: /bin/sh: nc: command not found +41ms
testcontainers [TRACE] [39f675b64955] Port check result exit code 1: /bin/bash: connect: Connection refused

But I noticed testcontainers:containers [39f675b64955] mysqld: Error writing file '/var/lib/mysql/auto.cnf' (OS errno 28 - No space left on device) +1ms, which didn't really make sense to me, as I have over 600GB available on my disk. So I removed all unused docker images, which amounted to ~60gb, using the $ docker system prune command - now it works!

I wasn't aware that Docker had a soft limit; a "Virtual disk limit" of 64 GB, which can be found under Settings -> Resources -> Virtual disk limit.

So now I know!

@mike12345567 mike12345567 mentioned this issue Dec 16, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mejenborg @gustavotrapp