Trouble Shooting

[Ying1123]

• Triton Kernel Fix:
  If you see No such file or directory: '/root/.triton/cache/e3457c918521f16104a655b081235e5a.....
  (issue caused by pytorch dependency of triton==2.3.0)
  1. You can fix it by hacking the file compiler.py

  vim /usr/local/lib/python3.10/dist-packages/triton/compiler/compiler.py
  
  L230
  
  self.asm = {
    file.suffix[1:]: file.read_bytes() if file.suffix[1:] == driver.binary_ext else None
  

  2. Or you can uninstall triton and reinstall the triton nightly

  pip uninstall -y triton triton-nightly
  pip install -U --index-url https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/Triton-Nightly/pypi/simple/ triton-nightly
  

[hnyls2002]

It is related to triton-lang/triton#2688. And there are some attempts to fix it. triton-lang/triton#3544 has been merged into triton==2.3.1 but bugs still remain. The latest triton-lang/triton#4295 seems to have solved it.

While waiting for triton==3.0.0 to apply this fix, I will try to fix it like vllm-project/vllm#6140 temporarily.