ORB-SLAM3 Stereo Mode Issue with Right Camera Not Detected

[OgulcanAtmaca]

Hi there,

I am trying to run the ORB_SLAM3 algorithm with the Oak-D-Pro camera from Luxonis. When I run ORB_SLAM3 in Stereo or Stereo_Inertial mode, there is no image from the right camera, only the image from the left camera. I have tried almost everything in this process but to no avail.
I got help from the following website as a source. https://qiita.com/nindanaoto/items/20858eca08aad90b5bab. I installed on a fresh Ubuntu and did not use the LXD container.
I calibrated the IMU and camera using "Kalibr".
I used the following version of depthai as I used the Noetic version of Ros.
https://github.com/luxonis/depthai-ros/releases/tag/v2.10.3-noetic
In the first terminal run the following command:
roslaunch depthai_examples stereo_inertial_node.launch
Here I have made some minor changes to the stereo_inertial_node.launch file. And I'll put the launch file below
`

<arg name="camera_model"         default="OAK-D"            /> <!-- 'zed' or 'zedm' -->
<arg name="tf_prefix"            default="oak"              />
<arg name="mode"                 default="stereo"            />
<arg name="base_frame"           default="oak-d_frame"      />
<arg name="parent_frame"         default="oak-d-base-frame" /> 
<arg name="imuMode"              default="1"                />   <!-- 0 -> COPY, 1 -> LINEAR_INTERPOLATE_GYRO, 2 -> LINEAR_INTERPOLATE_ACCEL -->

<arg name="cam_pos_x"             default="0.0" /> <!-- Position respect to base frame (i.e. "base_link) -->
<arg name="cam_pos_y"             default="0.0" /> <!-- Position respect to base frame (i.e. "base_link) -->
<arg name="cam_pos_z"             default="0.0" /> <!-- Position respect to base frame (i.e. "base_link) -->
<arg name="cam_roll"              default="0.0" /> <!-- Orientation respect to base frame (i.e. "base_link) -->
<arg name="cam_pitch"             default="0.0" /> <!-- Orientation respect to base frame (i.e. "base_link) -->
<arg name="cam_yaw"               default="0.0" /> <!-- Orientation respect to base frame (i.e. "base_link) -->

<arg name="lrcheck"               default="true" /> 
<arg name="extended"              default="false"/> 
<arg name="subpixel"              default="true" />
<arg name="rectify"               default="false" />
<arg name="depth_aligned"         default="false" />
<arg name="manualExposure"        default="false"/>
<arg name="expTime"               default="20000"/>
<arg name="sensIso"               default="800" />

<arg name="enableSpatialDetection" default="true" />
<arg name="syncNN"                 default="true" />
<arg name="detectionClassesCount"  default="80"   />
<arg name="nnName"                 default="x"         />
<arg name="resourceBaseFolder"     default="$(find depthai_examples)/resources" />

<!-- <arg name="nnPath"                 default="" /> -->
 <!-- see https://docs.luxonis.com/projects/api/en/latest/components/nodes/stereo_depth/#currently-configurable-blocks for possible combination of optionf for higher fps-->

<arg name="stereo_fps"            default="30"  />
<arg name="confidence"            default="200" />
<arg name="LRchecktresh"          default="5"   />
<arg name="monoResolution"        default="720p"/>

<arg name="rgbResolution"           default="1080p"/>
<arg name="rgbScaleNumerator"       default="2"/>
<arg name="rgbScaleDinominator"     default="3"/>
<arg name="previewWidth"            default="416"/>
<arg name="previewHeight"           default="416"/>

<arg name="angularVelCovariance"  default="0"   />
<arg name="linearAccelCovariance" default="0"   />

<arg name="enableDotProjector"    default="false"/>
<arg name="enableFloodLight"      default="false"/>
<arg name="dotProjectormA"        default="200.0"/>
<arg name="floodLightmA"          default="200.0"/>
<arg name="enableRviz"            default="false"/>
<arg name="enableMarkerPublish"   default="false" />

<arg name="enableRosBaseTimeUpdate" default="false" />  <!-- Used to handle when ROS time shifts and steady_clock does not -->

<include file="$(find depthai_descriptions)/launch/urdf.launch">
    <arg name="base_frame"      value="$(arg  base_frame)"  />
    <arg name="parent_frame"    value="$(arg  parent_frame)"/>
    <arg name="camera_model"    value="$(arg  camera_model)"/>
    <arg name="tf_prefix"       value="$(arg  tf_prefix)" />
    <arg name="cam_pos_x"       value="$(arg  cam_pos_x)"   />
    <arg name="cam_pos_y"       value="$(arg  cam_pos_y)"   />
    <arg name="cam_pos_z"       value="$(arg  cam_pos_z)"   />
    <arg name="cam_roll"        value="$(arg  cam_roll)"    />
    <arg name="cam_pitch"       value="$(arg  cam_pitch)"   />
    <arg name="cam_yaw"         value="$(arg  cam_yaw)"     />
</include>

<!-- launch-prefix="xterm -e gdb (add [- - args] without space) -->
<node name="stereo_inertial_publisher" pkg="depthai_examples" type="stereo_inertial_node" output="screen" required="true">
    <param name="mxId"                  value="$(arg mxId)"/>
    <param name="usb2Mode"              value="$(arg usb2Mode)"/>
    <param name="poeMode"               value="$(arg poeMode)"/>

    <param name="tf_prefix"             value="$(arg tf_prefix)"/>
    <param name="mode"                  value="$(arg mode)"/>
    <param name="imuMode"               value="$(arg imuMode)"/>

    <param name="lrcheck"               value="$(arg lrcheck)"/>
    <param name="extended"              value="$(arg extended)"/>
    <param name="subpixel"              value="$(arg subpixel)"/>
    <param name="rectify"               value="$(arg rectify)" />
    <param name="depth_aligned"         value="$(arg depth_aligned)" />
    <param name="manualExposure"        value="$(arg manualExposure)" />
    <param name="expTime"               value="$(arg expTime)" />
    <param name="sensIso"               value="$(arg sensIso)" />
    
    <param name="enableSpatialDetection"   value="$(arg enableSpatialDetection)" />
    <param name="syncNN"                   value="$(arg syncNN)" />
    <param name="detectionClassesCount"    value="$(arg detectionClassesCount)" />
    <param name="nnName"                   value="$(arg nnName)"/>
    <param name="resourceBaseFolder"       value="$(arg resourceBaseFolder)"/>

    <param name="stereo_fps"            value="$(arg stereo_fps)" />
    <param name="confidence"            value="$(arg confidence)" />
    <param name="LRchecktresh"          value="$(arg LRchecktresh)" />
    <param name="monoResolution"        value="$(arg monoResolution)" />

    <param name="rgbResolution"         value="$(arg rgbResolution)" />
    <param name="rgbScaleNumerator"     value="$(arg rgbScaleNumerator)" />
    <param name="rgbScaleDinominator"   value="$(arg rgbScaleDinominator)" />
    <param name="previewWidth"          value="$(arg previewWidth)" />
    <param name="previewHeight"         value="$(arg previewHeight)" />

    <param name="angularVelCovariance"  value="$(arg angularVelCovariance)" />
    <param name="linearAccelCovariance" value="$(arg linearAccelCovariance)" />

    <param name="enableDotProjector"    value="$(arg enableDotProjector)" />
    <param name="enableFloodLight"      value="$(arg enableFloodLight)" />
    <param name="dotProjectormA"        value="$(arg dotProjectormA)" />
    <param name="floodLightmA"          value="$(arg floodLightmA)" />
    <param name="enableRosBaseTimeUpdate" value="$(arg enableRosBaseTimeUpdate)" />
</node>            

<node pkg="nodelet" type="nodelet" name="nodelet_manager"  args="manager" output="screen"/>

<node pkg="nodelet" type="nodelet" name="depth_image_convertion_nodelet"
    args="load depth_image_proc/convert_metric nodelet_manager">
    <remap from="image_raw" to="/stereo_inertial_publisher/stereo/depth"/>    
    <remap from="image" to="/stereo_inertial_publisher/stereo/image"/>
</node>


<node if="$(eval arg('depth_aligned') == true)" pkg="nodelet" type="nodelet" name="depth_image_to_rgb_pointcloud"
    args="load depth_image_proc/point_cloud_xyzrgb nodelet_manager">
    <param name="queue_size"          value="10"/>

    <remap from="rgb/camera_info" to="stereo_inertial_publisher/color/camera_info"/>
    <remap from="rgb/image_rect_color" to="stereo_inertial_publisher/color/image"/>
    <remap from="depth_registered/image_rect" to="stereo_inertial_publisher/stereo/image"/>    
    <remap from="depth_registered/points" to="stereo_inertial_publisher/stereo/points"/>
</node>

<node if="$(eval arg('depth_aligned') == false)" pkg="nodelet" type="nodelet" name="depth_image_to_pointcloud"
    args="load depth_image_proc/point_cloud_xyz nodelet_manager">
    <param name="queue_size"          value="10"/>
    <remap from="camera_info" to="stereo_inertial_publisher/stereo/camera_info"/>
    <remap from="image_rect" to="stereo_inertial_publisher/stereo/image"/>    
    <remap from="points" to="stereo_inertial_publisher/stereo/points"/>
</node>


<node if="$(eval arg('enableMarkerPublish') == true)" type="markerPublisher.py" name="markerPublisher" pkg="depthai_examples">
    <remap from="spatialDetections" to="stereo_inertial_publisher/color/yolov4_Spatial_detections"/>
    <remap from="spatialDetectionMarkers" to="stereo_inertial_publisher/color/spatialDetectionMarkers"/>
</node>

` I turned off rectify here because I wanted to do the corrections with image_raws via stereo_image_proc. After running the command, my rostopic lists are as follows

/joint_states
/nodelet_manager/bond
/rosout
/rosout_agg
/stereo_inertial_publisher/imu
/stereo_inertial_publisher/left/camera_info
/stereo_inertial_publisher/left/image_raw
/stereo_inertial_publisher/left/image_raw/compressed
/stereo_inertial_publisher/left/image_raw/compressed/parameter_descriptions
/stereo_inertial_publisher/left/image_raw/compressed/parameter_updates
/stereo_inertial_publisher/left/image_raw/compressedDepth
/stereo_inertial_publisher/left/image_raw/compressedDepth/parameter_descriptions
/stereo_inertial_publisher/left/image_raw/compressedDepth/parameter_updates
/stereo_inertial_publisher/left/image_raw/theora
/stereo_inertial_publisher/left/image_raw/theora/parameter_descriptions
/stereo_inertial_publisher/left/image_raw/theora/parameter_updates
/stereo_inertial_publisher/right/camera_info
/stereo_inertial_publisher/right/image_raw
/stereo_inertial_publisher/right/image_raw/compressed
/stereo_inertial_publisher/right/image_raw/compressed/parameter_descriptions
/stereo_inertial_publisher/right/image_raw/compressed/parameter_updates
/stereo_inertial_publisher/right/image_raw/compressedDepth
/stereo_inertial_publisher/right/image_raw/compressedDepth/parameter_descriptions
/stereo_inertial_publisher/right/image_raw/compressedDepth/parameter_updates
/stereo_inertial_publisher/right/image_raw/theora
/stereo_inertial_publisher/right/image_raw/theora/parameter_descriptions
/stereo_inertial_publisher/right/image_raw/theora/parameter_updates
/stereo_inertial_publisher/stereo/disparity
/stereo_inertial_publisher/stereo/image
/stereo_inertial_publisher/stereo/image/compressed
/stereo_inertial_publisher/stereo/image/compressed/parameter_descriptions
/stereo_inertial_publisher/stereo/image/compressed/parameter_updates
/stereo_inertial_publisher/stereo/image/compressedDepth
/stereo_inertial_publisher/stereo/image/compressedDepth/parameter_descriptions
/stereo_inertial_publisher/stereo/image/compressedDepth/parameter_updates
/stereo_inertial_publisher/stereo/image/theora
/stereo_inertial_publisher/stereo/image/theora/parameter_descriptions
/stereo_inertial_publisher/stereo/image/theora/parameter_updates
/stereo_inertial_publisher/stereo/points
/tf
/tf_static

In the second terminal, run the following command
ROS_NAMESPACE=/stereo_inertial_publisher/ rosrun stereo_image_proc stereo_image_proc

And in the third terminal
rosrun ORB_SLAM3 Stereo_Inertial ORB_SLAM3/Vocabulary/ORBvoc.txt oak-d-params.yaml false /camera/left/image_raw:=/stereo_inertial_publisher/left/image_raw/camera/left/ /camera/right/image_raw:=/stereo_inertial_publisher/right/image_raw /imu:=/stereo_inertial_publisher/imu

My yaml file looks like this:

%YAML:1.0
Camera.type: "PinHole"

# Camera calibration and distortion parameters (OpenCV) 
# Copied from the EEPROM data dump in depthai_demo.py.
Camera.fx: 800.78991699
Camera.fy: 800.78991699
Camera.cx: 631.30053711
Camera.cy: 358.42175293

Camera.k1: 0.0
Camera.k2: 0.0
Camera.k3: 0.0
Camera.k4: 0.0
Camera.p1: 0.0
Camera.p2: 0.0

Camera2.fx: 794.44360352
Camera2.fy: 794.44360352
Camera2.cx: 655.11322021
Camera2.cy: 373.01904297

Camera2.k1: 0.0
Camera2.k2: 0.0
Camera2.k3: 0.0
Camera2.k4: 0.0
Camera2.p1: 0.0
Camera2.p2: 0.0

Camera.bFishEye: 0

Camera.width: 1280
Camera.height: 720

# Camera frames per second 
Camera.fps: 30.0

# stereo baseline times fx
Camera.bf: 60.05924377425

# Color order of the images (0: BGR, 1: RGB. It is ignored if images are grayscale)
Camera.RGB: 0

# Close/Far threshold. Baseline times.
ThDepth: 1.0

Stereo.ThDepth: 1.0
Stereo.b :0.075
Stereo.T_c1_c2: !!opencv-matrix
rows: 4
cols: 4
dt: f
data: [0.999993322285057, -0.00021991550145797296, 0.0036478791732720097, -0.07490939478841775,
0.00021600775474166326, 0.9999994025096745, 0.001071597377944539, 4.009736498019653e-06,
-0.003648112654574233, -0.0010708022519329299, 0.9999927723021768, 0.0012789063587460613,
0.0, 0.0, 0.0, 1.0]

# Transformation from body-frame (imu) to left camera

Tbc: !!opencv-matrix
rows: 4
cols: 4
dt: f
data: [0.006819407848809685, 0.999885118634947, -0.01353680940871299, 0.07058081582228529,
0.9999349374956549, -0.006694708786144591, 0.00923588922776827, 0.010005677849785306,
0.009144203199321436, -0.013598911965482609, -0.9998657175547137, 0.0054905640985514715,
0.0, 0.0, 0.0, 1.0]


# Specifies if the system inserts KeyFrames when the visual tracking is lost but the inertial tracking is alive.

IMU.InsertKFsWhenLost: 1 

#IMU Noise

# IMU noise
IMU.NoiseGyro: 0.00052310365
IMU.NoiseAcc: 0.00636068558
IMU.GyroWalk: 0.00008091795
IMU.AccWalk: 0.00061553137
IMU.Frequency: 250

#--------------------------------------------------------------------------------------------
# ORB Parameters
#--------------------------------------------------------------------------------------------

# ORB Extractor: Number of features per image
ORBextractor.nFeatures: 3000

# ORB Extractor: Scale factor between levels in the scale pyramid 
ORBextractor.scaleFactor: 1.4

# ORB Extractor: Number of levels in the scale pyramid 
ORBextractor.nLevels: 10

# ORB Extractor: Fast threshold
# Image is divided in a grid. At each cell FAST are extracted imposing a minimum response.
# Firstly we impose iniThFAST. If no corners are detected we impose a lower value minThFAST
# You can lower these values if your images have low contrast 
ORBextractor.iniThFAST: 20
ORBextractor.minThFAST: 7


#--------------------------------------------------------------------------------------------
# Viewer Parameters
#--------------------------------------------------------------------------------------------
Viewer.KeyFrameSize: 0.05
Viewer.KeyFrameLineWidth: 1
Viewer.GraphLineWidth: 0.9
Viewer.PointSize:2
Viewer.CameraSize: 0.08
Viewer.CameraLineWidth: 3
Viewer.ViewpointX: 0
Viewer.ViewpointY: -0.7
Viewer.ViewpointZ: -1.8
Viewer.ViewpointF: 500

Although I have written the necessary parameters for two cameras in the yaml file, when I run ORB_SLAM3, a text like the one below appears and only the left camera works. I am also attaching a screenshot of this.

Input sensor was set to: Stereo

Loading ORB Vocabulary. This could take a while...
Vocabulary loaded!

Initialization of Atlas from scratch 
Creation of new map with id: 0
Creation of new map with last KF id: 0
Seq. Name: 

Camera Parameters: 
- Camera: Pinhole
- Image scale: 1
- fx: 800.79
- fy: 800.79
- cx: 631.301
- cy: 358.422
- k1: 0
- k2: 0
- p1: 0
- p2: 0
- k3: 0
- fps: 30
- color order: BGR (ignored if grayscale)

Depth Threshold (Close/Far Points): 0.075

ORB Extractor Parameters: 
- Number of Features: 3000
- Scale Levels: 10
- Scale Factor: 1.4
- Initial Fast Threshold: 20
- Minimum Fast Threshold: 7
There are 1 cameras in the atlas
Camera 0 is pinhole
Starting the Viewer
First KF:0; Map init KF:0
New Map created with 289 points
TRACK_REF_KF: Less than 15 matches!!
Fail to track local map!
LM: Active map reset recieved

LM: Active map reset, waiting...
LM: Reseting current map in Local Mapping...
LM: End reseting Local Mapping...
LM: Reset free the mutex
LM: Active map reset, Done!!!
mnFirstFrameId = 0
mnInitialFrameId = 0
342 Frames set to lost
First KF:8; Map init KF:0
New Map created with 259 points

Thank you very much in advance for your help.