Shrinking Llama training to suite one GPU

[mahmoodn]

Hi,
Is it possible to run Llama training on 1 GPU for a test? I have tested with smaller sequence length and batch size of 1, but it seems that due to using Deepspeed in distributed_type: DEEPSPEED, it has to be a multi-node configuration. I can not find any other option other than DEEPSPEED.

Any idea about that?

[mahmoodn]

I have tried all possible options based on the suggestions on the web, e.g. increasing lora_size but it seems that 70b version needs 4 gpus.
I tried the same dataset with 7b and I was able to run it on single GPU with batch size of 1, 2, 4 and 8.