Docker operations timeout as part of docker-compose kill

[adrianclay]

Example console output:

16:57:58 ERROR: for hinge2eteststestagainstba77p3bhvw3oskbl52jigdubrdsdhd5krjrobmdmc3s4ixvgcj2a_router-api_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
16:57:58 
16:57:58 ERROR: for hinge2eteststestagainstba77p3bhvw3oskbl52jigdubrdsdhd5krjrobmdmc3s4ixvgcj2a_draft-router_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
16:57:58 
16:57:58 ERROR: for hinge2eteststestagainstba77p3bhvw3oskbl52jigdubrdsdhd5krjrobmdmc3s4ixvgcj2a_finder-frontend_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
16:57:58 
16:57:58 ERROR: for hinge2eteststestagainstba77p3bhvw3oskbl52jigdubrdsdhd5krjrobmdmc3s4ixvgcj2a_draft-collections_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
16:57:58 
16:57:58 ERROR: for hinge2eteststestagainstba77p3bhvw3oskbl52jigdubrdsdhd5krjrobmdmc3s4ixvgcj2a_content-tagger_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
16:57:58 An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
16:57:58 If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).
16:57:58 make: *** [kill] Error 1

When this happens as part of the teardown of Jenkins it won't fail the build but will mean the containers are in a partially reaped state.

Basic internet searching into what is going on has proved inconclusive. A deeper dive into what the docker daemon is blocking on with strace could identify the source of the slowness.

Increasing COMPOSE_HTTP_TIMEOUT could help. However, some have said that this only causes more waiting, and for docker to still timeout.

Running make stop straight after the error locally works fine without slowness which suggests docker is deadlocking. 😡

[kevindew]

Yeah I've seen this locally a few times and it doesn't feel like something where a timeout increase could help just seems to be the whole thing has clunked to a halt.