vision model initialization

[wangli68]

When I train vision model，I find the "text_loss" is "nan" ,and the "vision_loss" is "nan", the "loss" is "nan".
Why did this situation occur?
Is it because I didn't use a language model for initialization？
How to initialize visual models using language models? Do you want to configure any parameters?
{
'dataset_accumulated_tps': 51858.20817735159,
'dataset_average_tps': array(3.82700721e-05),
'dataset_example_index': 234,
'dataset_file_loc': 378520,
'dataset_total_tokens': 65536,
'gradient_norm': array(nan, dtype=float32),
'learning_rate': array(1.5999998e-05, dtype=float32),
'loss': array(nan, dtype=float32),
'param_norm': array(nan, dtype=float32),
'step': 0,
'text_acc': array(0., dtype=float32),
'text_loss': array(nan, dtype=float32),
'vision_acc': array(6.633284e-05, dtype=float32),
'vision_loss': array(nan, dtype=float32)
}
step:1
训练步数计数器step: 33% 1/3 [08:27<13:18, 399.38s/it]