Different results on MMdetection3D and OpenPCDet for DAIR-V2X-I training

[swwwu]

Hi there. I have a question about the evaluation results. I am trying to train PointPillars based on the DAIR-V2X-I dataset with MMdetection3D and OpenPCDet. I got results similar to those of the authors of DAIR-V2X while testing on MMdetection3D. (My results on MMdetection3D: Car-3D AP(R11-iou0.5): 63.58, 54.49, 54.49, DAIR-V2X paper: 63.07, 54.00, 54.01.) However, the result on OpenPCDet is much higher than that on MMdetetion3D. (My results on OpenPCDet: Car-3D AP(R11-iou0.5): 96.67, 90.05, 90.05.) I am very confused about this. Hope someone can help me find out the answer. Thank you very much.

Result on MMdetection3D:
----------- AP11 Results ------------
Car [email protected], 0.70, 0.70:
bbox AP11:24.1993, 19.3548, 19.5493
bev AP11:63.4588, 54.3793, 54.3846
3d AP11:63.0491, 53.9908, 53.9994
aos AP11:12.69, 10.15, 10.25
Car [email protected], 0.50, 0.50:
bbox AP11:24.1993, 19.3548, 19.5493
bev AP11:63.5912, 54.5022, 54.5028
3d AP11:63.5819, 54.4903, 54.4911
aos AP11:12.69, 10.15, 10.25
----------- AP40 Results ------------
Car [email protected], 0.70, 0.70:
bbox AP40:22.6050, 18.6080, 18.7952
bev AP40:67.2324, 54.7689, 54.7698
3d AP40:66.6725, 51.8965, 51.8953
aos AP40:11.85, 9.75, 9.85
Car [email protected], 0.50, 0.50:
bbox AP40:22.6050, 18.6080, 18.7952
bev AP40:67.4173, 54.9247, 54.9263
3d AP40:67.3984, 54.9069, 54.9088
aos AP40:11.85, 9.75, 9.85

Result on OpenPCDet:

Car [email protected], 0.70, 0.70:
bbox AP:27.9997, 24.0742, 24.2758
bev AP:96.2945, 89.6361, 89.6418
3d AP:86.9864, 78.0029, 78.0308
aos AP:14.65, 12.59, 12.70
Car [email protected], 0.70, 0.70:
bbox AP:27.4314, 23.6478, 23.8782
bev AP:97.8024, 92.4339, 92.4493
3d AP:89.8588, 81.2604, 81.2895
aos AP:14.35, 12.37, 12.49
Car [email protected], 0.50, 0.50:
bbox AP:27.9997, 24.0742, 24.2758
bev AP:97.0797, 90.5134, 90.5106
3d AP:96.6720, 90.0508, 90.0580
aos AP:14.65, 12.59, 12.70
Car [email protected], 0.50, 0.50:
bbox AP:27.4314, 23.6478, 23.8782
bev AP:98.8514, 95.7512, 95.7589
3d AP:98.4157, 94.9547, 94.9802
aos AP:14.35, 12.37, 12.49

[bbzh]

Hi @swwwu, did you figure out this? Can you share your experiences?