Is Layer Normalization in TransformerEncoderLayer2D placed in the correct position?

[S-aiueo32]

Hello,

I found Layer Normalization is placed after self-attention and the residual connection in TransformerEncoderLayer2D.

vedastr/vedastr/models/bodies/sequences/transformer/unit/encoder.py

Lines 60 to 63 in d617764

	attn_out, _ = self.attention(src, src, src, src_mask)
	out1 = src + attn_out
	out1 = out1.transpose(1, 2).contiguous().view(b, c, h, w)
	out1 = self.norm(self.attention_norm, out1)

I think it is a common way of basic transformers, however, the official implementation of SATRN places it before. Do you have any plans to align your code to the original one, or basis for reliability?

[ChaseMonsterAway]

Yes, we implement satrn before the code released. We will update the satrn in the future.

[S-aiueo32]

Good! I will keep this issue open for future fix about it!

[Johnson-yue]

this bug is not fixed for now ？？？？

[ChaseMonsterAway]

this bug is not fixed for now ？？？？

We do some other jobs for now, i will fix it as soon as possible.

[Johnson-yue]

ok，thanks