issues on dst

[smartyfh]

Hi,

when evaluating the JGA for DST, did you remove both the none slot and dontcare slot?

When I ran the dialogue_generation.py, it seems that the generated belief states are always empty in the MODEL_OUTPUT file. so could you please provide more details about how the model is trained for DST?

Thanks!

[ehosseiniasl]

Hi,

We are only removing none. it is fixed now.
On dialogue_generation.py, this seems a parsing issue. We will fix this

[fasterbuild]

@smartyfh empty belief issue: in generate_dialogue.py file, add text=text.strip() before tokenizer.encode(text), or there is always an space at the end. However I didn't get the joint acc in the paper, will you?

[smartyfh]

@smartyfh empty belief issue: in generate_dialogue.py file, add text=text.strip() before tokenizer.encode(text), or there is always an space at the end. However I didn't get the joint acc in the paper, will you?

@fasterbuild Have you checked all the checkpoints or just one checkpoint? If ignoring both none and dontcare slots, the results should be reproducible. However, if keeping the dontcare slots, the acc would go down several points. But this needs the author to confirm.

[gungui98]

@smartyfh I am struggling in reproducing the result, could you please share the hyper-params you use for the training?

[ShaneTian]

@smartyfh empty belief issue: in generate_dialogue.py file, add text=text.strip() before tokenizer.encode(text), or there is always an space at the end. However I didn't get the joint acc in the paper, will you?

@fasterbuild Have you checked all the checkpoints or just one checkpoint? If ignoring both none and dontcare slots, the results should be reproducible. However, if keeping the dontcare slots, the acc would go down several points. But this needs the author to confirm.

If keeping the dontcare slots, what is the JGA you get?

[HuangLK]

@smartyfh empty belief issue: in generate_dialogue.py file, add text=text.strip() before tokenizer.encode(text), or there is always an space at the end. However I didn't get the joint acc in the paper, will you?

@fasterbuild Have you checked all the checkpoints or just one checkpoint? If ignoring both none and dontcare slots, the results should be reproducible. However, if keeping the dontcare slots, the acc would go down several points. But this needs the author to confirm.

If keeping the dontcare slots, what is the JGA you get?

The JGA is 50.32% if keeping the dontcare. After ignoring both none and dontcare, I can achieve 55.45% JGA.

[libing125]

I got 50.46 joint accuracy, keeping dontcare and doing default cleaning.