TUM datasets not rectified correctly

[GoosebumpsFactory]

First of all thank you for publishing this package. In my lab we have been using the stereo_undistort_node in our pipeline for rectifying stereo-pairs calibrated using Kalibr and it has been of great help.

Recently we came across the TUM dataset:
https://vision.in.tum.de/data/datasets/visual-inertial-dataset
And we decided to run some experiments on the dataset. We found our pipeline performing poorly on the TUM dataset and further debugging has narrowed the problem down to rectified images failing the visual inspection test (there is a y-offset between left/right image features; also lines in the rectified right image are not perfectly straight).

Here is an example rectification result for a TUM data:

Here are the corresponding raw images:

The above rectification was generated using the following raw bag:
http://vision.in.tum.de/tumvi/calibrated/512_16/dataset-corridor4_512_16.bag
and the following calibration file:
https://vision.in.tum.de/_media/data/datasets/visual-inertial-dataset/pinhole-equi-512.zip

I have tried different scenes from the TUM dataset with similar rectification issues. I have also tried generating my own Kalibr file using the calibration bag they provide and there is no improvement in the rectification quality (though Kalbr reports sub pixel accuracy in calibrating the dataset). I used the following bag to generate my Kalibr file:
http://vision.in.tum.de/tumvi/calibrated/512_16/dataset-calib-cam4_512_16.bag

There are two possibilities I can think of that might cause this rectification issue:

1. TUM dataset published wrong calibration data that goes along with their dataset
2. There is a bug in the stereo_undistort_node that is exposed by this dataset.

I was hoping someone here (@ZacharyTaylor) might be willing to try stereo_undistort_node's performance on the TUM dataset. If there is a bug in the stereo_undistort_node you folks would be best at debugging it. Thanks.

[ZacharyTaylor]

Had a go at tracking this issue down, and while I can confirm it's a bug I am still looking for exactly what is going on.

The issue is definitely to do with how the rotation aligning the two cameras is calculated / applied (https://github.com/ethz-asl/image_undistort/blob/master/src/undistorter.cpp#L87)

The rotation is done in the undistorted image coordinates and it appears that this step is the source of the straight lines not being straight in the output. Applying it to the distorted vector solves this issue but we still see large vertical offsets in positions between the images at the edges. I don't have enough time to look through it in more detail now but may get some time to take a look next week.

[GoosebumpsFactory]

Thanks @ZacharyTaylor. Much appreciated.