Demo encountering Error

[abhiag719]

I get the following error when trying to run the demo code:

Traceback (most recent call last):
  File "demo.py", line 284, in <module>
    detector.run()
  File "demo.py", line 254, in run
    res = self.model.inference_single_image(cur_img.cuda().float(), (self.dataloader.seq_h, self.dataloader.seq_w), track_instances)
  File "/root/miniconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/h1/abhishek/blog/MOTR/models/motr.py", line 586, in inference_single_image
    track_instances=track_instances)
  File "/h1/abhishek/blog/MOTR/models/motr.py", line 515, in _forward_single_image
    hs, init_reference, inter_references, enc_outputs_class, enc_outputs_coord_unact = self.transformer(srcs, masks, pos, track_instances.query_pos, ref_pts=track_instances.ref_pts)
  File "/root/miniconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/h1/abhishek/blog/MOTR/models/deformable_transformer_plus.py", line 162, in forward
    memory = self.encoder(src_flatten, spatial_shapes, level_start_index, valid_ratios, lvl_pos_embed_flatten, mask_flatten)
  File "/root/miniconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/h1/abhishek/blog/MOTR/models/deformable_transformer_plus.py", line 266, in forward
    output = layer(output, pos, reference_points, spatial_shapes, level_start_index, padding_mask)
  File "/root/miniconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/h1/abhishek/blog/MOTR/models/deformable_transformer_plus.py", line 237, in forward
    src = self.forward_ffn(src)
  File "/h1/abhishek/blog/MOTR/models/deformable_transformer_plus.py", line 225, in forward_ffn
    src2 = self.linear2(self.dropout2(self.activation(self.linear1(src))))
  File "/root/miniconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/miniconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/root/miniconda3/envs/deformable_detr/lib/python3.7/site-packages/torch/nn/functional.py", line 1612, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

I have been running your demo in my docker with the following configuration :

FROM nvidia/cuda:9.2-cudnn7-devel-ubuntu18.04

RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6  -y

[dkhanna511]

Hi @abhiag719 , I have been facing the same error while evaluating the model on MOT17 using the script "r50_motr_submit.sh".
Were you able to solve this particular issue? I have tried several CUDA and Pytorch installations but I dont think that it's a CUDA error. Help me out if you've solved this