stereo_undistort not working with T265 camera

[mindThomas]

I am trying to use the stereo_undistort node together with the Intel Realsense T265 camera.
Although I have corrected the camera_info output to include the baseline as part of the Projection matrix, the initialization of the rectified image intrinsics fails, see: IntelRealSense/realsense-ros#1242

The code appears to be stuck in the for loop within setOptimalOutputCameraParameters since the resolution never converges and just gets bigger and bigger. When it finally quits due to it exceeding the number of iterations the resolution is so be that the undistortion map generation (i.e. remap within OpenCV) crashes.

I have tried to limit the number of iterations in setOptimalOutputCameraParameters to just 1, leading to a reasonable resolution. In that case the undistortion appears to work for both left and right camera, although the right image is flipped upside down?

As a side note, why is the rectification parameters recomputed each time a new image is received (as part of the image callback calling setInputCameraParameters --> generateRectificationParameters --> setOptimalOutputCameraParameters)?

[wang-yu-yang]

Sorry to disturb you, I modified the programs base_realsense_node.cpp and base_realsense_node.h according to your github, but when running Stereo_image_undistort node to rectify the image of T265, I encountered a problem, the corrected image cannot be displayed using rqt_image_view, and it runs The process is very slow. How do you display the corrected image? I don’t quite understand what you mentioned above. @mindThomas can you help me?

[mindThomas]

Hi @wang-yu-yang.
To try the changes that I've made which does indeed rectify the image but still has several flaws in terms of intrinsic initialization and the image flipping that I mentioned above, you need to do the following:

1. Checkout the expose-stereo-extrinsics branch from my realsense-ros fork: https://github.com/mindThomas/realsense-ros/tree/expose-stereo-extrinsics
2. Checkout the t265_stereo_undistort branch from my image_undistort fork: https://github.com/mindThomas/image_undistort/tree/t265_stereo_undistort
3. Compile the packages (e.g. with catkin build) and source the workspace.
4. Launch the realsense node with: roslaunch realsense2_camera rs_t265.launch
5. Launch the stereo_undistort node with: roslaunch image_undistort stereo_undistort_t265.launch
6. Now you can view all the images (including the rectified on the topics /camera/rect/first/image and /camera/rect/second/image) using e.g.: rosrun rqt_image_view rqt_image_view

[wang-yu-yang]

Thank you very much for your help. I have corrected the camera according to the above steps. but I have a warning

[ WARN] [1592478445.420755520]: For given K, T and P ([K,[0;0;0]]*T != P), replacing K with corrected value

I think it’s because my camera didn’t get a good calibration, and when I used the method you said to rectified the image and used the image for SLAM, it showed that my camera baseline was -0.0638m. Later, I used the Kalibr calibration tool to perform the camera calibration. Calibration, but when the camera model and distortion model are pinhole+equidistant, the camera cannot complete the calibration, and the camera parameters are only obtained when it is changed to omni+rantan, but the camera parameters cannot be used.

this is camera parameters before undistort:
left camera :
height: 800
width: 848
distortion_model: "equidistant"
D: [-0.006377882789820433, 0.04462724179029465, -0.04292846843600273, 0.008144219405949116, 0.0]
K: [286.64471435546875, 0.0, 424.2685852050781, 0.0, 286.81768798828125, 395.5517883300781, 0.0, 0.0, 1.0]
R: [1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0]
P: [286.64471435546875, 0.0, 424.2685852050781, 0.0, 0.0, 286.81768798828125, 395.5517883300781, 0.0, 0.0, 0.0, 1.0, 0.0]

right camera:
height: 800
width: 848
distortion_model: "equidistant"
D: [-0.005091331899166107, 0.03906574845314026, -0.03618951886892319, 0.00569117208942771, 0.0]
K: [286.4248046875, 0.0, 420.4132080078125, 0.0, 286.53759765625, 400.1791076660156, 0.0, 0.0, 1.0]
R: [0.9999791979789734, -0.0019368588691577315, -0.00615761149674654, 0.0019265528535470366, 0.9999967813491821, -0.0016791693633422256, 0.006160843186080456, 0.0016672712517902255, 0.9999796748161316]
P: [289.0089416503906, 0.14617842435836792, 418.6409606933594, 18.297866821289062, 3.017470598220825, 287.2038879394531, 399.6898193359375, 0.0, 0.006160843186080456, 0.0016672712517902255, 0.9999796748161316, 0.0]

[jokla]

@mindThomas Which compiler are you using? Could be related to this issue? #44

[mindThomas]

@jokla I am indeed using GCC 7.4. I haven't tested this with a lower version so it is likely that it is related. Thank you.

[rakshith95]

@mindThomas were you able to fix this? I have the same issue where the right image is upside down for some reason and so the disparity is obviously nonsense.

[ryleymcc]

@mindThomas were you able to fix this? I have the same issue where the right image is upside down for some reason and so the disparity is obviously nonsense.

I have the same issue

[cggos]

I forked the image_undistort and update: https://github.com/cggos/image_undistort/tree/rs_t265, with which we can use the t265 camera but the point cloud is very poor