File /MixSort/yolox/mixsort_oc_tracker/mixformer.py:103 causes TypeError: expected Tensor as element 0 in argument 0, but got tuple

[SergeySandler]

torch.stack(template_imgs).float().div(255) on line 103 in /MixSort/yolox/mixsort_oc_tracker/mixformer.py

template_imgs = normalize(
torch.stack(template_imgs).float().div(255),
self.cfg.DATA.MEAN,
self.cfg.DATA.STD,
)

causes TypeError: expected Tensor as element 0 in argument 0, but got tuple.

I've followed the installation instructions https://github.com/MCG-NJU/MixSort#installation. My code fragment that triggers the problem is as following,

from collections import namedtuple
from yolox.mixsort_oc_tracker.mixsort_oc_tracker import MIXTracker

ArgsTuple = namedtuple("ArgsTuple", ["script", "config", "alpha", "radius", "mix_iou", "local_rank"])
argsTuple = ArgsTuple("mixformer_deit", "track", 0.6, 0, 0.2, 0)
tracker = MIXTracker(det_thresh = 0.5, args = argsTuple, use_byte=False)
vcapture = cv2.VideoCapture("path-to-video")
while(True):
    success, frame = vcapture.read()
    if not success:
        break
    detections = # fill in from a custom YOLOv8 model
    detections = tracker.update(raw_detection, (1,1), (1,1), frame, scale = False) 

The call stack:
File /MixSort/yolox/mixsort_oc_tracker/mixsort_oc_tracker.py:284, in MIXTracker.update(self, output_results, img_info, img_size, img, scale)
File /MixSort/yolox/mixsort_oc_tracker/association.py:272, in associate(detections, trackers, iou_threshold, velocities, previous_obs, vdc_weight, img, mixformer, alpha, templates)
File /MixSort/yolox/mixsort_oc_tracker/mixformer.py:103, in MixFormer.compute_vit_sim(self, detections, trackers, img, templates)

[ret-1]

Hi,

From the error message you provided, it seems that the template_imgs variable is not a list of torch.Tensor objects as expected. The error indicates that one or more elements in the template_imgs list are tuples, which is causing the torch.stack operation to fail.

To help diagnose the issue, could you please check the value and type of the template_imgs variable right before the normalize() function call? Specifically, the variable is assigned with the following code:

MixSort/yolox/mixsort_oc_tracker/mixformer.py

Line 90 in 52d317c

template_imgs=[templates[int(t[-1])] for t in trackers]

Please ensure that the templates list contains only torch.Tensor objects and that the indexing operation templates[int(t[-1])] always returns a tensor.

By examining the contents of template_imgs before the normalize() call, we should be able to identify the source of the problem.

[SergeySandler]

@ret-1, templates is a list of tuples, please see below, print statements were added after line 90.

        print(type(templates))           # <class 'list'>
        print(len(templates))            # 20
        print(type(templates[0]))        # <class 'tuple'>
        print(len(templates[0]))         # 2
        print(type(templates[0][0]))     # <class 'torch.Tensor'>
        print(len(templates[0][0]))      # 3
        print(templates[0][0].shape)     # torch.Size([3, 96, 96])
        print(type(templates[0][1]))     # <class 'list'>
        print(len(templates[0][1]))      # 0

Notice the first element in the list is a tuple of a tensor and an empty list.

[ret-1]

@SergeySandler From your details, the issue seems to originate from the exception handling in

MixSort/yolox/mixsort_oc_tracker/mixformer.py

Lines 66 to 72 in 52d317c

	try:
	resized_img = resized_crop(
	img, y, x, crop_sz, crop_sz, [output_sz, output_sz]
	)
	except: # too small box
	zero_img = torch.zeros((3, output_sz, output_sz)).cuda()
	return zero_img if annos is None else zero_img, []

This behavior is actually a simplified adaptation from the handling below where data is marked as invalid if the boxes are too small. The choice to return a zero tensor and an empty list was made because this specific case wasn't encountered during our experiments.

MixSort/MixViT/lib/train/data/processing.py

Lines 273 to 280 in 52d317c

	# 2021.1.9 Check whether data is valid. Avoid too small bounding boxes
	w, h = torch.stack(jittered_anno, dim=0)[:, 2], torch.stack(jittered_anno, dim=0)[:, 3]
	crop_sz = torch.ceil(torch.sqrt(w * h) * self.search_area_factor[s])
	if (crop_sz < 1).any():
	data['valid'] = False
	# print("Too small box is found. Replace it with new data.")
	return data

Could you confirm if it aligns with the issue you observed? If so, consider making the necessary adjustments and submitting a pull request to handle such cases more effectively. Thanks for pointing this out.

[sky-creater]

@SergeySandler Have you solved this problem? Could you provide the solution?

[SergeySandler]

@SergeySandler Have you solved this problem? Could you provide the solution?

@sky-creater no unfortunately not solved.