This repository has been archived by the owner on Jan 21, 2025. It is now read-only.

Does load-balanced loss help the loss converge？ #391

Open

mathfinder opened this issue Jun 13, 2023 · 0 comments

mathfinder commented Jun 13, 2023

mesh/mesh_tensorflow/transformer/moe.py

Line 935 in fbf7b1e

# LOAD BALANCING LOSS

I try load-balanced loss in my project and find load-balanced loss does not help loss converge.

Does it only balance the load, but does not help the loss convergence, or even slightly hurt the model?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.