Error with NVIDIA A100 80GB

[asoxien]

Hello everybody;

With NVIDIA A100 80GB GPU I get this error:
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
the some code was successfully run on a machine with NVIDIA GeForce GTX 1070
I use the latest rastervision master branch and the pytorch-latest image

Thank you

[AdeelH]

Try to rebuild the docker image on the new device. Clone the repo and run docker/build. This way, it might install PyTorch from a binary compatible with that GPU.

[asoxien]

Hi Adeel and thank you;
The same problem after building the docker image exactly the same error.
I also tried to build the image with argument CUDA_VERSION="11.1.1" but nothing changed
Still searching for a solution

[asoxien]

Hello again;
I finally resolved the problem by:
pip3 install torch==1.9.1+cu111 torchvision==0.10.1+cu111 torchaudio==0.9.1 -f https://download.pytorch.org/whl/torch_stable.html

but I can't verify if it's working nominally and optimally or not!!!

[ammarsdc]

Hi @AdeelH @lewfish @asoxien.

Here, we were actually able to run RV using Tesla T4 and 3090 Ti for quite sometimes now.
We initially using Tesla T4 and had no issue running any release of RV. But then, we tried running on 3090 Ti, we can't.

Fortunately, based on suggestion given in here, one of thread in Gitter and debugging made by one of my colleague @acun1994 we finally made it able to run.

Here were steps we took to solve it:

• Update build file in docker directory: Update CUDA_VERSION to 11.3.1.
• Update Dockerfile in repository base directory:
  • Set CUDNN from 7 to 8 at base docker image:

    # From this:
    nvidia/cuda:${CUDA_VERSION}-cudnn7-devel-ubuntu16.04
    
    # To this:
    nvidia/cuda:${CUDA_VERSION}-cudnn8-devel-ubuntu16.04
    

  • Add -f https://download.pytorch.org/whl/torch_stable.html at the end of line 56 or the end of line wherever pip install for package rastervision_pytorch_learner was.
  • Append these two lines after line 62 or after where all packages were pip installed:

    RUN pip uninstall torch torchvision torchaudio -y
    RUN pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu113
    

• Update requirements.txt file in rastervision_pytorch_learner directory:
  • Add --extra-index-url https://download.pytorch.org/whl/cu113 anywhere in the file.
  • Update version of PyTorch:

    # From this:
    torch==1.11.*
    torchvision==0.12.*
    
    # To this:
    torch==1.12.*
    torchvision==0.13.*
    

• Update utils.py in rastervision_pytorch_learner/rastervision/pytorch_learner/utils at line 7: from torch.hub import _import_module as import_module
• If error still occurred, update again Dockerfile by adding this command at the last line: RUN pip install pyOpenSSL --upgrade

Commit 72b54a9 Onwards

Today, as a lot of updates were done by RV team, We found some of the fixes highlighted above was already fixed.

Hence, we once again try git clone the latest repository, docker/build then docker/run, but 3090 Ti still cannot run the training with resulting error CUDA_LAUNCH_BLOCKING.

However, by adding this --extra-index-url https://download.pytorch.org/whl/cu113 in requirements.txt file in rastervision_pytorch_learner directory seem fixing this issue. No need to update PyTorch version like previous fixes since it was updated by team. Redocker/build and redocker/run is now makes RV works well as usual.

Hope this solution is helpful.