This repository has been archived by the owner on Jan 21, 2025. It is now read-only.

mask_1_flat and mask_2_flat applied to gates twice? #378

Open

marhlder opened this issue Mar 31, 2022 · 0 comments

marhlder commented Mar 31, 2022 •

edited

Loading

mesh/mesh_tensorflow/transformer/moe.py

Line 1694 in 6b31c0f

gate_1 * mask_1_flat

It should not cause any issues I guess. Just unnecessary computation?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.