[QUESTION] When will model have _extra_state?

[1049451037]

After updating to the main branch of Megatron-LM recently, I met this error when loading model:

Unexpected key(s) in state_dict: "decoder.layers.0.self_attention.core_attention._extra_state"

The checkpoint is transformed by the tools/checkpoint/convert.py, and loaded by pretrain_gpt.py.

[sambar1729]

I had faced this error too, recently, and had to write a downstream script to remove this from every layer -- looks like the Mcore format is out of sync with what MegatronLM consumes?