what is the meaning of num_warps?

[andy90]

I am very new to Triton and got confused about the meaning of num_warps. I think the default warp size for NVIDIA GPUs is 32. So the num_warps should just be BLOCK_SIZE divided by 32. Why num_warps is a tunable parameter in Triton?

[ptillet]

it's an optimization parameter. num_warps is the number of warps the compiler will parallelize work over. if num_warps=4 (default) then the kernel will be parallelized for 128 cuda threads