Why are the "wrong" sentences are learned during training via LM?

[fabiang7]

Maybe I don't interpret the model(...)-function correctly, but I see the following:

While training you put the correct and wrong rocstories together into the decoder. They both go through the embedding + decoder and then into the sparse_softmax_cross_entropy-function.

This means, though, that the model also learns to generate wrong sentences, or am I missing something?

My intuition would be to set all masks to 0 for the wrong sentences?!

Thanks and regards