Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Precompiled 535 driver can't be installed : image pull failure #682

Closed
easyrider14 opened this issue Mar 19, 2024 · 2 comments
Closed

Precompiled 535 driver can't be installed : image pull failure #682

easyrider14 opened this issue Mar 19, 2024 · 2 comments
Assignees
Labels
bug Issue/PR to expose/discuss/fix a bug

Comments

@easyrider14
Copy link

1. Quick Debug Information

  • OS/Version: Ubuntu22.04
  • Container Runtime Type/Version: Containerd
  • K8s Flavor/Version: K8s
  • GPU Operator Version: 23.9.0 and 23.9.2

2. Issue or feature description

nvidia-driver-daemonset tries to pull an image that does not exist
it tries to pull nvcr.io/nvidia/driver:535-5.15.0-100-generic-ubuntu22.04

I've been using version 23.9.0 of gpu-operator for a while now, using unmodified helm templates (usePrecompiled: true, version: 535)
It was working as expected until a few days ago. While installing a new cluster, nvidia-driver-daemonset failed to install driver
The daemonset was named nvidia-driver-daemonset-5.15.0-100-generic-ubuntu22.04. The image it tries to pull does not exist in nvidia's catalog, only tag 535-5.15.0-97-generic-ubuntu22.04 exists.
I've updated gpu operator from 23.9.0 to latest 23.9.2, but it still fails
I can make it work by setting usePrecompiled to false, and setting the complete driver version

@easyrider14 easyrider14 changed the title Preinstalled 535 driver can't be installed : image pull failure Precompiled 535 driver can't be installed : image pull failure Mar 20, 2024
@shivamerla
Copy link
Contributor

shivamerla commented Mar 21, 2024

@easyrider14 thanks for reporting this. We needed a small fix in the CI to address this. We will publish these images soon.

@shivamerla shivamerla self-assigned this Mar 21, 2024
@shivamerla shivamerla added the bug Issue/PR to expose/discuss/fix a bug label Mar 21, 2024
@shivamerla
Copy link
Contributor

This is now fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue/PR to expose/discuss/fix a bug
Projects
None yet
Development

No branches or pull requests

2 participants