[Bug]: fused_moe_kernel compile bug #6103
Labels
bug
Something isn't working
Comments
|
cc @njhill |
jeejeelee
changed the title
|
Yeah, we've fixed this issue on our fork (as you found here). Let me create a PR to contribute the fix upstream. |
|
I was able to work around this bug by building the latest Triton code from source. |
It seems that the triton main has updated the related code, I don't have more time to dive into this update. For now, I believe #6140 can serve as a temporary solution to address this issue. Once VLLM updates the triton version, we can revisit this issue again. |
|
@randxie Interesting. I actually tried to test these changes that were merged into Triton main in our fork, but it didn't help. I don't really see much else that has changed in the meantime (at least in Agree that getting the fix from Triton is the long-term solution though. |
|
@tdoublep I believe there were more changes involved. If you check the log above, Triton tried to load files with naming pattern as |
|
removing the entire triton cache dir every time before run vllm can be a temp workaround, when I run deepseekv2. |
7 tasks
ThomasRaoux
pushed a commit
to triton-lang/triton
that referenced
this issue
Jul 10, 2024
…4295) # Summary there've been multiple issues discussing around the `FileNotFoundError` on compilation when `CompiledKernel` is trying to read from the listed ASM files. #2688 #4002 vllm-project/vllm#6103 etc. and there have been some attempts to address it such as #3544 . This PR attempts to explain the root cause and suggest a fix. # Why When a kernel is being compiled, triton first writes IRs to triton cache dir ([ref](https://github.com/triton-lang/triton/blob/78091647fccb6825ed9956ff7c0300859856d261/python/triton/compiler/compiler.py#L289)). Inside of the write operation, the process first writes it to a temp file unique to the current process (plus a uuid to distinguish between multiple processes with same PID on different hosts sharing the same underlying FS) ([ref](https://github.com/triton-lang/triton/blob/c14b033cd979d5c39e5fdb3847c022fa5d71a0c1/python/triton/runtime/cache.py#L124-L130)) and then atomically `os.replace` it to the final file name. Afterwards the `CompiledKernel` lists all the IRs and reads them ([ref](https://github.com/triton-lang/triton/blob/78091647fccb6825ed9956ff7c0300859856d261/python/triton/compiler/compiler.py#L362-L367)). On multiprocess set up this may however result in a race condition. Let's focus on a case where there's one host with 2 processes on it.  At the time when `pid 1` lists ASMs, the dir may contain temp files generated from another process `pid 2`. However at the time when `pid 1` proceeds to read bytes from the listed files, `pid2` may have already `os.replace`ed its temp files, so `pid 1` will encounter `FileNotFoundError` when trying to read the temp file generated by `pid 2`. IBM/vllm#35 (comment) also believes this is the root cause. # How There're multiple potential solutions towards this, as mentioned in IBM/vllm#35 (comment) as well: - let each process write to a private temp dir instead so `glob` won't bother taking the temp stuff into consideration - or, exclude `tmp.pid_*` from `glob` This PR tries to go with the 1st approach to avoid adding an assumption on the tmp file pattern (which is only used in `runtime/cache.py`) in `compiler/compiler.py` but is open to any suggestion. Thanks! Complete the following tasks before sending your PR, and replace `[ ]` with `[x]` to indicate you have done them. - [x] I am not making a trivial change, such as fixing a typo in a comment. - [x] I have written a PR description following these [rules](https://cbea.ms/git-commit/#why-not-how). - [x] I have run `pre-commit run --from-ref origin/main --to-ref HEAD`. - Select one of the following. - [ ] I have added tests. - `/test` for `lit` tests - `/unittest` for C++ tests - `/python/test` for end-to-end tests - [x] This PR does not need a test because `not applicable`. - Select one of the following. - [x] I have not added any `lit` tests. - [ ] The `lit` tests I have added follow these [best practices](https://mlir.llvm.org/getting_started/TestingGuide/#filecheck-best-practices), including the "tests should be minimal" section. (Usually running Python code and using the instructions it generates is not minimal.)
|
There was a PR merged into Triton yesterday that tries to address this issue: triton-lang/triton#4295. This fix is not yet included in |
|
So I've been digging into this a bit more and here is a summary of my findings:
|
Firtly, I strongly agree with merging #6140. For |
|
Just for the context, I got the same bug with the latest vllm-server v0.5.1 on python -u -m vllm.entrypoints.openai.api_server --model mistralai/Mixtral-8x22B-Instruct-v0.1 --dtype auto --tensor-parallel-size 4 --gpu-memory-utilization 0.95 --swap-space 4 --download-dir /data/vllm-data --max-seq-len-to-capture 8192 --host 0.0.0.0 --port 8079 --max-model-len 16384Reproduced 6 out of 7 times on the first request to the server. |
|
Any updates on this? |
|
Fix #6140 is ready from my pov, will try to get it approved and merged asap. |
3 tasks
bertmaher
pushed a commit
to bertmaher/triton
that referenced
this issue
Dec 10, 2024
…riton-lang#4295) # Summary there've been multiple issues discussing around the `FileNotFoundError` on compilation when `CompiledKernel` is trying to read from the listed ASM files. triton-lang#2688 triton-lang#4002 vllm-project/vllm#6103 etc. and there have been some attempts to address it such as triton-lang#3544 . This PR attempts to explain the root cause and suggest a fix. # Why When a kernel is being compiled, triton first writes IRs to triton cache dir ([ref](https://github.com/triton-lang/triton/blob/78091647fccb6825ed9956ff7c0300859856d261/python/triton/compiler/compiler.py#L289)). Inside of the write operation, the process first writes it to a temp file unique to the current process (plus a uuid to distinguish between multiple processes with same PID on different hosts sharing the same underlying FS) ([ref](https://github.com/triton-lang/triton/blob/c14b033cd979d5c39e5fdb3847c022fa5d71a0c1/python/triton/runtime/cache.py#L124-L130)) and then atomically `os.replace` it to the final file name. Afterwards the `CompiledKernel` lists all the IRs and reads them ([ref](https://github.com/triton-lang/triton/blob/78091647fccb6825ed9956ff7c0300859856d261/python/triton/compiler/compiler.py#L362-L367)). On multiprocess set up this may however result in a race condition. Let's focus on a case where there's one host with 2 processes on it.  At the time when `pid 1` lists ASMs, the dir may contain temp files generated from another process `pid 2`. However at the time when `pid 1` proceeds to read bytes from the listed files, `pid2` may have already `os.replace`ed its temp files, so `pid 1` will encounter `FileNotFoundError` when trying to read the temp file generated by `pid 2`. IBM/vllm#35 (comment) also believes this is the root cause. # How There're multiple potential solutions towards this, as mentioned in IBM/vllm#35 (comment) as well: - let each process write to a private temp dir instead so `glob` won't bother taking the temp stuff into consideration - or, exclude `tmp.pid_*` from `glob` This PR tries to go with the 1st approach to avoid adding an assumption on the tmp file pattern (which is only used in `runtime/cache.py`) in `compiler/compiler.py` but is open to any suggestion. Thanks! Complete the following tasks before sending your PR, and replace `[ ]` with `[x]` to indicate you have done them. - [x] I am not making a trivial change, such as fixing a typo in a comment. - [x] I have written a PR description following these [rules](https://cbea.ms/git-commit/#why-not-how). - [x] I have run `pre-commit run --from-ref origin/main --to-ref HEAD`. - Select one of the following. - [ ] I have added tests. - `/test` for `lit` tests - `/unittest` for C++ tests - `/python/test` for end-to-end tests - [x] This PR does not need a test because `not applicable`. - Select one of the following. - [x] I have not added any `lit` tests. - [ ] The `lit` tests I have added follow these [best practices](https://mlir.llvm.org/getting_started/TestingGuide/#filecheck-best-practices), including the "tests should be minimal" section. (Usually running Python code and using the instructions it generates is not minimal.)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Your current environment
🐛 Describe the bug
Description
#5036 will raise the following error when using TP>1. Similar issues also persisted when I tested the Qwen1.5-MoE model
due to the use of the fused_moe_kernel triton kernel.
Possible solutions
I believe it's a triton bug. I then debugged the triton code and investigated the related issue. and found that the cause of this issue is well explained here. Until triton officially addresses this issue, we may need to resolve it in a similar manner. Another approach is to set
distributed_executor_backendtoraywhen using the triton kernel.Reproducible Code
The text was updated successfully, but these errors were encountered: