docker/moby image pull timeouts

[fposch]

Hi,

I know it is not strictly Rancher-related and the problem apparently lies in dockerd(moby) (as one of the two container engines) but I'm also getting this when trying to deploy my kubernetes cluster through Rancher Desktop

In events, I'm getting lots of these (image name obfuscated)

Failed to pull image "xxxxxx.dkr.ecr.eu-west-1.amazonaws.com/application-image:210630": rpc error: code = Unknown desc = context deadline exceeded

(I'm using Kubernetes v1.22.12 if that matters).

It seems to be an issue with dockerd/moby where pulling an image (for a pod/container) is aborted after 1 minute, so depending on your connection speed and image sizes you may never get a pod to start unless you pull the image manually so that it is already in cache while deploying.

Somebody has provided a fix for the engine (or presumably the runtime interface/CRI) in version 0.26.0 but it would require an upgrade in Rancher packages make use of that (I tried 1.6.1 and still having the same problem).

I hope that makes sense.

BR,
Florian

[fposch]

Thanks for getting this in the next release, I tried with the highly anticipated 1.7.0 release and I am unfortunately still getting these errors.

I'm running Rancher 1.7.0 on Windows with WSL integration.

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  16m                   default-scheduler  Successfully assigned my-rancher-desktop/service-584c5865f6-4l78j to desktop-evj6k82
  Normal   Pulling    8m52s (x4 over 16m)   kubelet            Pulling image "1234.dkr.ecr.eu-west-1.amazonaws.com/service:version-SNAPSHOT"
  Warning  Failed     7m32s (x4 over 13m)   kubelet            Failed to pull image "1234.dkr.ecr.eu-west-1.amazonaws.com/service:version-SNAPSHOT": rpc error: code = Unknown desc = context canceled
  Warning  Failed     7m32s (x4 over 13m)   kubelet            Error: ErrImagePull
  Warning  Failed     6m54s (x7 over 13m)   kubelet            Error: ImagePullBackOff
  Normal   BackOff    6m16s (x10 over 13m)  kubelet            Back-off pulling image "1234.dkr.ecr.eu-west-1.amazonaws.com/service:version-SNAPSHOT"