RuntimeError: The size of tensor a (147) must match the size of tensor b (293) at non-singleton dimension 3

[Vijaysivadas]

I get this error while running the inference.ipynb code in the docs folder. I did not change anything in the file. Does anyone know how to solve it ?

Thank you for your time.

RuntimeError Traceback (most recent call last)
Cell In[12], line 1
----> 1 inference(model, tokenizer, context_len, features, args)

Cell In[10], line 39
36 streamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
38 with torch.inference_mode():
---> 39 output_ids = model.generate(
40 input_ids,
41 images=image[None,].cuda(),
42 do_sample=True,
43 temperature=args.temperature,
44 max_new_tokens=1024,
45 streamer=streamer,
46 use_cache=True,
47 stopping_criteria=[stopping_criteria]
48 )
50 outputs = tokenizer.decode(output_ids[0, input_ids.shape[1]:]).strip()
51 conv.messages[-1][-1] = outputs

File c:\Users\Codes Easy\Documents\Vijay\ML\llm-projects\llm-projects\Lib\site-packages\torch\utils_contextlib.py:115, in context_decorator..decorate_context(*args, **kwargs)
112 @functools.wraps(func)
113 def decorate_context(*args, **kwargs):
114 with ctx_factory():
...
--> 137 expanded_attn_mask = causal_4d_mask.masked_fill(expanded_attn_mask.bool(), torch.finfo(dtype).min)
139 # expanded_attn_mask + causal_4d_mask can cause some overflow
140 expanded_4d_mask = expanded_attn_mask

RuntimeError: The size of tensor a (147) must match the size of tensor b (293) at non-singleton dimension 3

[huangb23]

You might want to check the versions of the relevant libraries, especially Transformers and PyTorch. Sometimes, version mismatches can lead to such errors.