Is triton suited for cases where the input tensor shape tensors?

[RuABraun]

This is my scenario. Not clear to me from the documentation if triton might not be suitable for when the batch and length dimensions are changing. I saw some references to the kernels having to be recompiled everytime the sizes change?

[binarman]

@RuABraun
Kernels are recompiled only once for every new size.
Compiled kernels are stored in cache (see $HOME/.triton/cache/)

If your data have finite number of shapes, eventually triton will have all code variants.
You can make several "warmup" runs before actual workload to get predictable latencies.

[RuABraun]

Okay great! (it is a finite number) Thank you

[ptillet]

Note: it isn't totally true that Kernels are recompiled for every new size. Kernels are re-compiled when (1) signature changes or (2) int kernel arguments become (or stop becoming) divisible by 16, or equal to 0, or equal to 1. If all your batch sizes are multiples of 16 and everything stays identical, you shouldn't have more than one kernel per op assuming all the dtypes stay the same