Ask for "practical tricks" for Hungarian loss

[sangwoomo]

Hi, it seems that the code use the average set loss (in "practical tricks" in Section 3.2) for the Chamfer loss but not for the Hungarian loss. Should I need to use the average one or only the final one to reproduce your bounding box and state prediction results?
Also, line 233-239 seems not used in the left of the code!

[Cyanogenoid]

To reproduce the results in the paper, you don't need to change anything in the code. The code is exactly what produced the results in the paper.

Whoops, lines 233 to 239 are indeed not used, well spotted! I was intending the bounding box and state prediction experiments to use the average set loss, but clearly I was using just the final set. The DSPN results should be better with the average set loss, but I'll have to try and see if the hyperparams still work.

[Cyanogenoid]

I tried a run where I used the average set loss with the Hungarian loss by making

dspn/dspn/train.py

Lines 240 to 242 in 2515f61

	set_loss = utils.hungarian_loss(
	progress[-1], target_set, thread_pool=pool
	).unsqueeze(0)

use a and b as originally intended.

The following single run result corresponds to Table 2 in the paper.

Box prediction	AP 50	AP 90	AP 95	AP 98	AP 99
DSPN (10 iters)	99.8%	99.3%	92.7%	18.0%	0.5%
DSPN (20 iters)	99.9%	99.6%	91.3%	14.5%	0.4%
DSPN (30 iters)	99.9%	99.6%	88.3%	12.2%	0.3%

AP 98 and AP 99 are worse, whereas AP 50, AP 90, and AP 95 are all better than in the paper.

The following single run result corresponds to Table 3 in the paper.

State prediction	AP inf	AP 1	AP 0.5	AP 0.25	AP 0.125
DSPN (10 iters)	79.6%	73.3%	59.3%	23.6%	3.2%
DSPN (20 iters)	83.6%	80.7%	69.9%	29.2%	3.9%
DSPN (30 iters)	85.0%	82.9%	72.9%	30.4%	4.0%

This time, almost all results are better than in the paper (especially AP 0.5 and AP 0.25) without changing any hyperparameters. More iterations also no longer makes some of the results worse, so the algorithm is simply more stable. Hooray!

[sangwoomo]

Thank you for the detailed response!
Great to see that fixing the bug even improves your reported results :)

[Cyanogenoid]

I'm updating the paper with the fixed results in an appendix soon.
@sangwoomo Are you okay with me crediting you for finding the bug?

[sangwoomo]

My pleasure :)