Unable to get extrinsics calibration between Avia and UHD DSLR

[VisionaryMind]

I have been running multiple tests these past few days, and I cannot get any working extrinsics calibration with our dataset. Please, if you will have a moment, please indicate what is wrong with these images:

We have 24 of them at 4, 7, and 8-meter distances, covering the entire FOV of both the Avia and our DSLR camera. The camera is generating hi-res images (above) at a resolution of 5184x2920. The images shown here have a .5 and 1-inchi white border, however, we have also used calibration boards no border. Neither produces results. Corners are found in the images but not in the point cloud. Here is one of the clouds:

There are more than enough points to perform RANSAC without loss of the small plane. I will be trying once again tonight, this time with the checkerboard positioned farther up on the stand so that it covers the top. I shouldn't think that the small point at the top would count as an "obstruction", but perhaps this is the problem.

[HViktorTsoi]

Hi, sorry for the late response,

1. There should be no extra borders around the checkerboard, so the checkerboard you are using now is inappropriate;
2. This tool requires reflectivity intensity from LiDAR, but in your point cloud visualization, we don’t see reflectivity information, is it the visualization problem or there is no reflectivity information from your LiDAR data?
3. The protruding part at the top of the checkerboard does affect the checkerboard detection, please remove that part, or let the checkerboard cover that part.

[VisionaryMind]

Hi, sorry for the late response,

1. There should be no extra borders around the checkerboard, so the checkerboard you are using now is inappropriate;

The checkerboard shown in the picture does have a border, but we have the same problem with this board as well:

2. This tool requires reflectivity intensity from LiDAR, but in your point cloud visualization, we don’t see reflectivity information, is it the visualization problem or there is no reflectivity information from your LiDAR data?

The provided image was generated by PCL Viewer, and it does not use the intensity data. The point cloud, however, does have intensity information, as it was generated by your calibration tool off the /livox/lidar topic. Here is a view with intensity info:

3. The protruding part at the top of the checkerboard does affect the checkerboard detection, please remove that part, or let the checkerboard cover that part.

Yes, we will try that next. I will report back with results.

[VisionaryMind]

I just repeated the data collection using this calibration board:

ACSC could not find either 2D or 3D corners. The following error message is displayed for every image-cloud pair:

Localization done. min cost=9.75709919708
Intensity pivot: 103.866622927
ERROR: Corners not found!!!
Re-sample time: 0.0217430591583s
Points after voxelize: 11263

Optimized loss: 48.7050764321
**ERROR: Corners not found!!!**

Calculating frame: 0 / 26
**Missing 2D corners, skip this pair**

A bit more information on this configuration:

1. Livox Avia is mounted about 16cm above a DSLR camera.
2. The DSLR camera has has a Leica lens extending about 10cm in front of its sensor and the Avia.
3. Image resolution is 5184x2920.

Here are the intrinsics and distortion used:

3645.87124432589 0.0 2612.23847215656
0 3646.18121592077 1336.94941902667
0 0 1

0.0 0.0 0.0 0.0 0.0

There were originally tangential and radial distortion values as calculated by MATLAB, but we have discovered the MATLAB is not capable of accurately calculating such values. In general, it produces unreliable intrinsics for camera calibration. I suspect that improper intrinsics could be causing the problem, but running the "findChessboardCorners" method in OpenCV separately is able to identity 2D corners without issue.

What could be wrong with our configuration? Is it just a matter of inaccurate intrinsics? In which case -- what is the recommended method for capturing such values for DSLR cameras. We have used both ROS camera calibration and MATLAB, and both produce different results--- and both lead to inaccurate extrinsics calculations.

[HViktorTsoi]

Hi, it can be seen from the log that, the main problem is missing 2D corners, (rather than missing 3D corners, the Optimized loss is < 100, means the 3D corner detection from point cloud is successful), it means that the problem is that opencv failed to detect corners from the images.

We use the new findChessboardCornersSB function in opencv, rather than findChessboardCorners, so you may try running the findChessboardCornersSB method separately and check if it is able to identity 2D corners.

Besides, checkerboard corner detection in opencv doesn't require camera intrinsic parameters, so the corner detection issue is not because of the intrinsic parameters. However, the camera intrinsic does have critical impact on the extrinsic calculation.

The problem is not easy to locate just through text, it's best that you can send the saved data pairs and intrinsic parameters to us and to debug. ([email protected])

If possible, also send the images that you used to calibrate the DSLR, we will check if the data satisfies the requirement of intrinsic calibration.

[VisionaryMind]

Thank you for offering to help troubleshoot this issue with us. I have sent you the full dataset, as requested. Indeed, when running findChessboardCorners and findChessboardCornersSB on these images, none of them are recognized. More specifically, the non-SB algorithm is able to find corners in 2-3 photos, but SB has not demonstrated this capability. Further, 3D corners are not recognized for 5 / 29 pairs.

[Leon-0919]

@VisionaryMind Hi bro, can I please have your data for validation as well? Really appreciate the help.