Memory bandwidth limit

[ManuelKessler]

Hello,

thanks for the very nice example and optimization guide. I do not fully understand your reasoning in part 1 to establish an upper bound for the memory bandwidth. Starting from kernel 1, we see an actual fetch size double the theoretical fetch size.
So the potential reduction in memory traffic is ((2014+1064)-(1074+1061))/(2014+1064)=30.6%. The corresponding potential speedup then might be 1/(1-0.306)=1.44, which is what you write as well, so we agree here. Keeping all other factors equal, you conclude an upper bound of the memory bandwidth of 1165 GB/s, targeting 71% of peak HBM bandwidth.
What I do not understand (besides the 44% reduced memory traffic, which is probably just a typo and should read 30.6%), is why you claim you can't get closer to 100% peak HBM bandwidth. After all, changing the kernel, as you do in the following parts, is just NOT "keeping all other factors equal". So one could hope to arrive at an implementation somewhat better than the 71%. Admittedly, this would be getting more and more difficult, but I do see no fundamental barrier at this point.
Perhaps you can elaborate a little more if and why the 71% are limiting in this case.
Thanks again for enlightening me (and probably many others).

Ciao,
Manuel

[jychang48]

Hi Manual,

Thanks for your response. To answer your questions:

1. Yes it should be 30.6% thanks for catching

2. Part 1 does not claim that we cannot get closer to 100% peak HBM bandwidth. The reported 1165 GB/s, which happens to be 71% of the HBM peak, is a loose target one should minimally achieve if reduction of the memory traffic is the sole optimization goal. In the subsequent parts, there are cases where the FOM exceeds 1165 GB/s. We have deliberately avoided making any strong performance claims, as achieving the highest possible FOM will depend on a variety of factors e.g., ROCm versions, system configurations, and other optimization tricks not covered in this Laplacian series. That said, "upper bound on effective ..." is not the right phrase to use, rather it's a "more realistic target for effective ..."

Hope this helps,
Justin

[ManuelKessler]

Thanks for the clarification. I understand your motivation, and it gives me more confidence I have grasped some basics about performance tuning on this device. Thanks!
Ciao,
Manuel