[QUESTION]how to incorporate MOE into hybrid mamba efficiently

[sunying2018]

how to incorporate MOE into hybrid mamba efficiently
Hi,

You provided code pretrain_mamba.py contains the mamba spec to contain transformer, mamba and mlp layers. I'm wondering if there is an easier way to incorporate MOE to replace some of MLP layers? Since I found args.spec conflicts with simply specified moe args setting. What considerations require these two to be set as mutually exclusive? Thank you so much!