About the results of this model

[qingyue2014]

Hi, I rerun this project with the same experimental setting. But the joint accuracy only achieves around 55% on multiwoz2.1 dataset. Is there any special version of the transformers or something?

[smartyfh]

Hi, I rerun this project with the same experimental setting. But the joint accuracy only achieves around 55% on multiwoz2.1 dataset. Is there any special version of the transformers or something?

You can try the checkpoint I provided

[qingyue2014]

I try the checkpoint you provided and it achieves the result paper reporting. But I rerun the project several times still can't gain the best model(more than one percent drop). Would you recheck that the hyper-param settings shown in the GitHub is completely same to that in provided best model ? Thx.

[smartyfh]

I try the checkpoint you provided and it achieves the result paper reporting. But I rerun the project several times still can't gain the best model(more than one percent drop). Would you recheck that the hyper-param settings shown in the GitHub is completely same to that in provided best model ? Thx.

The parameters are the same. The performance could be affected by the GPUs you used.

[AlbertChen1991]

I was also unable to reproduce the result, only get 54.3 on MultiWOZ 2.1 dataset

[qingyue2014]

I was also unable to reproduce the result, only get 54.3 on MultiWOZ 2.1 dataset

I get 55.04 on 2.1 hhh

[smartyfh]

I was also unable to reproduce the result, only get 54.3 on MultiWOZ 2.1 dataset

I get 55.04 on 2.1 hhh

Thanks for your interest. Please check the training records here: https://github.com/smartyfh/DST-STAR/blob/main/out-bert/exp/exp.txt

[couragelfyang]

@smartyfh
Thanks for your interesting work. I've compare https://github.com/smartyfh/DST-STAR/blob/main/out-bert/exp/exp.txt and the results reported in your paper. Is the "slot_acc" same to the "Appendix A slot-specific Accuracy"? Why are they different in results.

[smartyfh]

@smartyfh
Thanks for your interesting work. I've compare https://github.com/smartyfh/DST-STAR/blob/main/out-bert/exp/exp.txt and the results reported in your paper. Is the "slot_acc" same to the "Appendix A slot-specific Accuracy"? Why are they different in results.

The reported results are the average of multiple random seeds. So is the joint goal accuracy.

[Janeey99]

i got 55.79% on multiwoz2.1 with random seed=42, and i only change batch_size to 8, other params are same as the author's. i dont know why.........

[smartyfh]

i got 55.79% on multiwoz2.1 with random seed=42, and i only change batch_size to 8, other params are same as the author's. i dont know why.........

Hi, thank you for the feedback, batch size can indeed affect model performance, it is good to know that the model works when setting the batch size to 8.