Add a triaging guide for suspected rocMLIR failures.

[manupak]

This PR adds a doc to help migraphx developers
triage a suspected rocMLIR failure.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.20%. Comparing base (d9367cb) to head (8ab1604).
Report is 148 commits behind head on develop.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #3227   +/-   ##
========================================
  Coverage    92.20%   92.20%           
========================================
  Files          493      493           
  Lines        19700    19700           
========================================
  Hits         18164    18164           
  Misses        1536     1536           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[umangyadav]

Another debugging check could be reverting rocMLIR SHA.
If there was a known working SHA before try building MIGraphX against that and see if that works without any problems.
and then try newer rocMLIR SHA and see if it fails or passes with MIGraphX. That could indicate if problem lies in rocMLIR or MIGraphX

[krzysz00]

Yeah

But also, conversely, holding the rocMLIR SHA constant at the newest one while debugging can be useful

[manupak]

changing rocMLIR SHA feels like something rocMLIR team should be doing...
I m not sure whether it is advisable to try older SHAs where bugs are fixed in the later ones.

Therefore, I think disabling rocMLIR might be enough with the latest SHA in most cases to see whether its rocMLIR vs MIGraphX bug. (Yes, there can be complex cases where rocMLIR offloads creating a different graph structure exposing a MIGraphX bug but with current infrastructure there is no good way to handle that -- we can have a separate discussion how to improve debuggability)

I d go further and say unless there is a warning from the following, we should not be changing rocMLIR SHA in debugging.

AMDMIGraphX/src/targets/gpu/mlir.cpp

Lines 42 to 43 in 3e65032

	#if !defined(MLIR_MIGRAPHX_DIALECT_API_VERSION) || MLIR_MIGRAPHX_DIALECT_API_VERSION != 4
	#warning "Incompatible version of rocMLIR library used, disabling"

[migraphx-bot]

Test	Batch	Rate new cab23d	Rate old 497c27	Diff	Compare
torchvision-resnet50	64	1,749.69	1,748.90	0.05%	✅
torchvision-resnet50_fp16	64	4,078.30	4,080.75	-0.06%	✅
torchvision-densenet121	32	1,471.95	1,467.80	0.28%	✅
torchvision-densenet121_fp16	32	2,520.20	2,527.69	-0.30%	✅
torchvision-inceptionv3	32	889.21	888.98	0.03%	✅
torchvision-inceptionv3_fp16	32	1,483.71	1,484.24	-0.04%	✅
cadene-inceptionv4	16	412.28	412.13	0.04%	✅
cadene-resnext64x4	16	419.25	419.33	-0.02%	✅
slim-mobilenet	64	4,004.68	4,004.86	-0.00%	✅
slim-nasnetalarge	64	100.95	100.96	-0.02%	✅
slim-resnet50v2	64	1,678.44	1,678.94	-0.03%	✅
bert-mrpc-onnx	8	615.65	616.38	-0.12%	✅
bert-mrpc-tf	1	277.65	279.52	-0.67%	✅
pytorch-examples-wlang-gru	1	326.51	326.38	0.04%	✅
pytorch-examples-wlang-lstm	1	294.10	294.48	-0.13%	✅
torchvision-resnet50_1	1	471.91	472.05	-0.03%	✅
cadene-dpn92_1	1	246.78	246.45	0.13%	✅
cadene-resnext101_1	1	204.36	203.78	0.28%	✅
onnx-taau-downsample	1	205.75	206.10	-0.17%	✅
dlrm-criteoterabyte	1	22.89	22.90	-0.05%	✅
dlrm-criteoterabyte_fp16	1	42.72	42.66	0.14%	✅
agentmodel	1	6,117.63	7,432.29	-17.69%	🔴
unet_fp16	2	34.28	34.31	-0.07%	✅
resnet50v1_fp16	1	595.91	578.62	2.99%	✅
resnet50v1_int8	1	572.92	577.30	-0.76%	✅
bert_base_cased_fp16	64	645.75	645.72	0.00%	✅
bert_large_uncased_fp16	32	198.84	198.85	-0.01%	✅
bert_large_fp16	1	117.45	117.34	0.10%	✅
distilgpt2_fp16	16	1,211.95	1,210.51	0.12%	✅
yolov5s	1	301.25	293.94	2.49%	✅
tinyllama	1	23.32	23.30	0.09%	✅
vicuna-fastchat	1	132.99	133.83	-0.62%	✅
whisper-tiny-encoder	1	244.20	243.92	0.11%	✅
whisper-tiny-decoder	1	255.72	256.31	-0.23%	✅

This build is not recommended to merge 🔴

[migraphx-bot]

     ✅ bert-mrpc-onnx: PASSED: MIGraphX meets tolerance

     ✅ bert-mrpc-tf: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-gru: PASSED: MIGraphX meets tolerance

     ✅ pytorch-examples-wlang-lstm: PASSED: MIGraphX meets tolerance

     ✅ torchvision-resnet50_1: PASSED: MIGraphX meets tolerance

     ✅ cadene-dpn92_1: PASSED: MIGraphX meets tolerance

     ✅ cadene-resnext101_1: PASSED: MIGraphX meets tolerance

     ✅ dlrm-criteoterabyte: PASSED: MIGraphX meets tolerance

     ✅ agentmodel: PASSED: MIGraphX meets tolerance

     ✅ unet: PASSED: MIGraphX meets tolerance

     ✅ resnet50v1: PASSED: MIGraphX meets tolerance

     ✅ bert_base_cased_fp16: PASSED: MIGraphX meets tolerance

🔴bert_large_uncased_fp16: FAILED: MIGraphX is not within tolerance - check verbose output

     ✅ bert_large: PASSED: MIGraphX meets tolerance

     ✅ yolov5s: PASSED: MIGraphX meets tolerance

     ✅ tinyllama: PASSED: MIGraphX meets tolerance

     ✅ vicuna-fastchat: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-encoder: PASSED: MIGraphX meets tolerance

     ✅ whisper-tiny-decoder: PASSED: MIGraphX meets tolerance

     ✅ distilgpt2_fp16: PASSED: MIGraphX meets tolerance

[krzysz00]

Yeah

But also, conversely, holding the rocMLIR SHA constant at the newest one while debugging can be useful

[krzysz00]

I don't think MIGraphX is able to do this easily

[manupak]

yes but someone has to do it, right ?

[umangyadav]

https://github.com/ROCm/AMDMIGraphX/pull/3182/files#

With this PR it would be possible to dump MIGraphX programs for each module as MXR and then migraphx-driver can driver verify them.

[CharlieL7]

minor comments only

[manupak]

Thanks all for reviewing and more importantly suggesting changes in GH interface :)