ValueError: rank 1 not found in topology

[BAAI-WuDao]

When I run bash ``examples/gpt/pretrain_gpt_distributed.sh"

It reports the information

and reports this error

I follows the errors, it seems that the problem is located in topology.py, Line 43,

  for global_rank, coord in enumerate(cartesian_product(*ranges)):
      key = {axis: coord[self.axes.index(axis)] for axis in self.axes}
      key = self.ProcessCoord(**key)
      # for example, {ProcessCoord(row=0, col=1) : 1}
      self.mapping[key] = global_rank

for I print the variable self.mapping at topology.py, Line 131, it is empty.

[gongwei-130]

Hi WUDAO, could you provide your experiment set up? Like parameters used in pretrain_gpt_distributed.sh, how many machines and how many GPUs per machine?

[BAAI-WuDao]

Hi， the pretrain_gpt_distributed.sh is set up as

#! /bin/bash

Runs the "345M" parameter model

DATA_PATH='/data/wang/models/Sailing/examples/gpt2'
CHECKPOINT_PATH='./'

export WORKER_0_HOST=localhost
export WORKER_0_PORT=6000
export NUM_WORKER=1
export WORKER_RANK=0
export GPU_PER_WORKER=4

MASTER_PORT=6002
MASTER_ADDR=$WORKER_0_HOST

GPUS_PER_NODE=$GPU_PER_WORKER

NNODES=$NUM_WORKER
NODE_RANK=$WORKER_RANK

WORLD_SIZE=$(($GPUS_PER_NODE*$NNODES))

base_dir=$(cd dirname $0; pwd)
echo base_dir $base_dir

DISTRIBUTED_ARGS="--nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT"

ds_config='{
"train_micro_batch_size_per_gpu":16,
"train_batch_size" : 16,
"gradient_accumulation_steps": 2,
"steps_per_print": 1,
"gradient_clipping": 1.0,
"zero_optimization": {
"stage": 0,
"allgather_partitions": true,
"allgather_bucket_size": 500000000,
"overlap_comm": true,
"reduce_scatter": true,
"reduce_bucket_size": 500000000,
"contiguous_gradients" : true,
"cpu_offload": false
},
"fp16": {
"enabled": true,
"loss_scale": 0,
"loss_scale_window": 1000,
"hysteresis": 2,
"min_loss_scale": 1
},
"wall_clock_breakdown": true
}'

python3 -m torch.distributed.launch $DISTRIBUTED_ARGS
--no_python --use_env python3
${base_dir}/pretrain_gpt2.py
--model-parallel-size 2
--num-stages 2
--num-layers 24
--hidden-size 1024
--train-batch-size 64
--gradient_accumulation_steps 16
--num-attention-heads 16
--batch-size 4
--seq-length 1024
--max-position-embeddings 1024
--train-iters 500000
--lr-decay-iters 450000
--save $CHECKPOINT_PATH
--load $CHECKPOINT_PATH
--data-path $DATA_PATH/my-gpt2_text_document
--vocab-file $DATA_PATH/gpt2-vocab.json
--merge-file $DATA_PATH/gpt2-merges.txt
--data-impl mmap
--split 949,50,1
--distributed-backend nccl
--lr 0.00025
--lr-decay-style cosine
--min-lr 1.0e-5
--weight-decay 1e-2
--clip-grad 1.0
--warmup .02
--log-interval 1
--save-interval 100000
--vocab-size 145608
--DDP-impl torch
--eod-mask-loss
--deepspeed-pipeline
--deepspeed
--config_param "$ds_config"
--fp16
--partition_method "type:ParallelTransformerLayerPiped"
$@
set +x

[gongwei-130]

Hi WuDao,

Sorry, I am not able to repo your issue with provided setting. I suggest do following two steps to exclude some possible issue:

1. Add following into your pretrain_gpt_distributed.sh

       export WORKER_0_HOST=127.0.0.1
       export DMLC_NODE_HOST=127.0.0.1
       export BYTEPS_WITH_UCX=0 
       export DMLC_ENABLE_UCX=0
       export DMLC_ENABLE_RDMA=0

2. Try setting up environment with following docker file, I will push this docker file into repo soon.

FROM nvcr.io/nvidia/pytorch:21.05-py3 
RUN pip3 install boto3 regex tensorboardX==1.8 wheel pybind11 ninja psutil pyprof
RUN apt-get -yq autoremove --purge ibverbs-providers
RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends --allow-downgrades \
     libibverbs-dev=28.0-1ubuntu1 libibverbs1=28.0-1ubuntu1

RUN apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends --allow-downgrades \
        cmake \
        libopenmpi-dev \
        openmpi-bin \
        openssh-client \
        openssh-server \
        ibverbs-providers \
        libibverbs-dev=28.0-1ubuntu1 \
        librdmacm-dev \
        vim \
        iputils-ping \
        llvm-10-dev \
        iproute2 \
        unzip

RUN ln -s /usr/bin/aclocal-1.16 /usr/local/bin/aclocal-1.14
RUN ln -s /usr/bin/automake /usr/local/bin/automake-1.14

ENV LD_LIBRARY_PATH "/usr/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}"
ENV BYTEPS_WITH_UCX 0

RUN pip3 install https://giant-model-package.tos-cn-beijing.volces.com/byteps-0.7.2-cp38-cp38-linux_x86_64.whl
WORKDIR /root

Hi， the pretrain_gpt_distributed.sh is set up as

#! /bin/bash

Runs the "345M" parameter model

DATA_PATH='/data/wang/models/Sailing/examples/gpt2' CHECKPOINT_PATH='./'

export WORKER_0_HOST=localhost export WORKER_0_PORT=6000 export NUM_WORKER=1 export WORKER_RANK=0 export GPU_PER_WORKER=4

MASTER_PORT=6002 MASTER_ADDR=$WORKER_0_HOST

GPUS_PER_NODE=$GPU_PER_WORKER

NNODES=$NUM_WORKER NODE_RANK=$WORKER_RANK

WORLD_SIZE=$(($GPUS_PER_NODE*$NNODES))

base_dir=$(cd dirname $0; pwd) echo base_dir $base_dir

DISTRIBUTED_ARGS="--nproc_per_node $GPUS_PER_NODE --nnodes $NNODES --node_rank $NODE_RANK --master_addr $MASTER_ADDR --master_port $MASTER_PORT"

ds_config='{ "train_micro_batch_size_per_gpu":16, "train_batch_size" : 16, "gradient_accumulation_steps": 2, "steps_per_print": 1, "gradient_clipping": 1.0, "zero_optimization": { "stage": 0, "allgather_partitions": true, "allgather_bucket_size": 500000000, "overlap_comm": true, "reduce_scatter": true, "reduce_bucket_size": 500000000, "contiguous_gradients" : true, "cpu_offload": false }, "fp16": { "enabled": true, "loss_scale": 0, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1 }, "wall_clock_breakdown": true }'

python3 -m torch.distributed.launch $DISTRIBUTED_ARGS --no_python --use_env python3 ${base_dir}/pretrain_gpt2.py --model-parallel-size 2 --num-stages 2 --num-layers 24 --hidden-size 1024 --train-batch-size 64 --gradient_accumulation_steps 16 --num-attention-heads 16 --batch-size 4 --seq-length 1024 --max-position-embeddings 1024 --train-iters 500000 --lr-decay-iters 450000 --save $CHECKPOINT_PATH --load $CHECKPOINT_PATH --data-path $DATA_PATH/my-gpt2_text_document --vocab-file $DATA_PATH/gpt2-vocab.json --merge-file $DATA_PATH/gpt2-merges.txt --data-impl mmap --split 949,50,1 --distributed-backend nccl --lr 0.00025 --lr-decay-style cosine --min-lr 1.0e-5 --weight-decay 1e-2 --clip-grad 1.0 --warmup .02 --log-interval 1 --save-interval 100000 --vocab-size 145608 --DDP-impl torch --eod-mask-loss --deepspeed-pipeline --deepspeed --config_param "$ds_config" --fp16 --partition_method "type:ParallelTransformerLayerPiped" $@ set +x

[BAAI-WuDao]

Thank you very much for you response!
I follow the steps, but the bugs still exists.
Can you give a example, that I can set the value of ``mapping" at line 136 in topology.py

[gongwei-130]

Thank you very much for you response! I follow the steps, but the bugs still exists. Can you give a example, that I can set the value of ``mapping" at line 136 in topology.py

Hi WuDao,

Could you paste your full log here? Don't use screen shot, so that I can search text.