[REQUEST] Some questions about deepspeed sequence parallel

[yingtongxiong]

Hello, I want to run sequence parallel on pure deepspeed repo. However, I found that it is necessary to let developer to create sequence parallel process group, is it right? I want to know there is any solutions to use sequence parallel or MoE(which also requires expert_data_process_group and so on) on pure deepspeed.

[samadejacobs]

@yingtongxiong , the recommended use of Deepspeed sequence parallelism (deepspeed ulysses) is to call it from a client framework/script. Please take a look at these two examples: Megatron-DeepSpeed, HugingFace transformer

[yingtongxiong]

Okay Thank you very much

[yingtongxiong]

https://github.com/microsoft/DeepSpeedExamples/blob/uly-hf/post_training/sequence_parallelism/test_ulysses.py#L113 I see in here, the mesh_param is commented, so I think if I want to use sp, this parameters should be transmitted, is it right? @samadejacobs

[yingtongxiong]

Aslo, when I use sp all2all overlap, I found a little bug.

DeepSpeed/deepspeed/sequence/layer.py

Line 242 in a1b0c35

assert ctx.stream != None

when stream is not None, the assert is still False, so I think it should be "ctx.stream is not None".

[ronald-d-rogers]

@yingtongxiong I think mesh_param is not needed as it is inferred from the config dict parameters (i.e. sequence_parallel_size/data_parallel_size).

[yingtongxiong]

@yingtongxiong I think mesh_param is not needed as it is inferred from the config dict parameters (i.e. sequence_parallel_size/data_parallel_size).

Thank you, And I don't know how to set sequence_parallel_size? In config? how to transmit to deepspeed.initialize? maybe can you give me an example?

[ronald-d-rogers]

Sure, in the test it is here:
https://github.com/microsoft/DeepSpeedExamples/blob/uly-hf/post_training/sequence_parallelism/test_ulysses.py#L57

data_parallel_size is typically num_gpus // sequence_parallel_size, where num_gpus is total gpus (or num you want to use).

Note that the number of processes you want to launch is still the number of gpus you are using, so data_parallel_size * sequence_parallel_size.

So, for example, if you're using torchrun to run the script then you want --nproc-per-node to be set to 4 if you have 4 total gpus, whatever your arrangement of sp and dp is (so long as sp*dp=4).

Each data parallel group takes in a separate copy of the full train dataset, and each sequence parallel rank in each data parallel group needs to be given the slice of data according to that rank. You need to do this splitting yourself. You must also let the model know which parts of the sequence it's processing by providing a position_ids tensor, which is a tensor of values 0 ... seq_len, split according to rank.

AFAIK, the above test_ulysses.py example does not work without a PR to Transformers that has not yet been merged:
huggingface/transformers#32305

There is another attempt at integrating ulysses requiring changes across accelerate and transformers made by @zeyugao here:
huggingface/accelerate#2877
huggingface/transformers#31525

It seems likely that the changes to modeling_llama.py made in the above PR would also be needed.

The main question I have is if we need to use this new loss if we're using tranformers, and would that entail updating all of the trainers.

[ronald-d-rogers]

@yingtongxiong: I will try to put together a working example soon (if what I have is actually working)

[yingtongxiong]

to

Thank you very much

[yingtongxiong]

@yingtongxiong: I will try to put together a working example soon (if what I have is actually working)

okay thank you