Replicating results

[mattiadurso]

Following the discussions in #74 and #67, I ran COLMAP and GLOMAP on the (distorted) ETH3D MVS (DSLR) dataset. However, I’m struggling to reproduce your results and am encountering two main issues:

1. High variability between different runs (AUC scores can be from 0 or 90).
2. Consistently obtaining AUC scores around 20 points lower than your reported average.

Since I’m using the AUC function directly from PxSfm (as mentioned in #67), I’m wondering if you tested the models with any non-default parameters. Some potential factors I’ve considered include:

• Random seed (for feature extraction, matching, and mapping)
• Image reconstruction seed ID
• Whether intrinsics were passed manually
• (BA) intrinsics refinement

Any insight or advice on how to address these issues would be greatly appreciated.

Thank you for your time.

[lpanaf]

Hi, regarding the experiments, we have fed in the ground truth intrinsic, so that intrinsics are properly shared among images, and are initialized with nice values. Also, we use undistorted images, instead of raw ones. Have you obtained reasonable values for COLMAP as a starting point?

[mattiadurso]

we have fed in the ground truth intrinsic, [...] we use undistorted images

That was the point! Thanks. Now I get mostly the same scores as in the paper.

It's interesting that in some scenes I get better results (for both models) than the GLOMAP paper. Also, at least in one run I did, GLOMAP does not fail on exhibition hall and scores 84.6 AUC while COLMAP 84.5 (starting from the same database). Also old computer scores better on my run of GLOMAP while the COLMAP run it's the same of yours.

If you are interest we can discuss about it.

Thanks again!

[ahojnnes]

@mattiadurso Great that you managed to replicate our experiments. Some smaller bugs have been fixed since submission of the paper and some components of the pipeline have been improved. So, it is expected that the latest commits in the main branch will produce slightly different (hopefully always better :-)) results than reported in the paper. The same goes for COLMAP.

[lpanaf]

Hi @mattiadurso, great to hear that you reproduce the result. Just an additional note, the major difference between the current version and the result in the paper should be the initialization of rotation averaging. Now, I enabled initialization from the maximum spanning tree, which solved the problem for exhibition_hall by chance. If you are interested, you may try disabling this function (there is a flag called skip_initialization in the options of rotation averaging, then the result should be more similar to what we have in the paper :)

[mattiadurso]

Thanks a lot for answering me! Hope to meet you at ECCV next week!

[w-hc]

hi would you mind sharing the colmap feature extractor command you used to populate the intrinsics with gt? I have so far failed to reproduce the numbers in Table 3.

More strangely, for [electro, kicker, lounge] the number of images in glomap output images.bin is different from the number of provided images.

As an example, these are the steps I took on "lounge".

colmap feature_extractor \
    --image_path    ./raw/lounge/images/dslr_images_undistorted \
    --database_path ./colmap_db/lounge.db \
    --ImageReader.camera_model PINHOLE  \
    --ImageReader.camera_params "3425.17,3423.46,3118.1,2072.96"

colmap exhaustive_matcher --database_path ./colmap_db/lounge.db

glomap mapper \
    --image_path ./raw/lounge/images/dslr_images_undistorted \
    --database_path ./colmap_db/lounge.db \ 
    --output_path ./results/lounge

I am using the cam parameter numbers in lounge/dslr_calibration_undistorted/cameras.txt.

In the resulting .bin file I could only read 6 images, but there should be 10.

[mattiadurso]

If the intrinsics changes after you ran the model, try is to set the intrinsics equal within the same folder and divided the images in folders according to the intrinsics. Tell the model to do not refine the intrinsics during BA.

If the final model has less images than in the original folder, it may have failed to register them.

[w-hc]

Thanks for your help @mattiadurso
Do you observe registration failure on [electro, kicker, lounge]?

[w-hc]

@lpanaf Thanks for giving us glomap. Do you mind providing more details on reproducing the results?

[lpanaf]

Hi @w-hc, sorry for the late reply. I checked my result, and indeed, for electro and lounge, not all images are registered. You can also see in the table in the paper that the AUC @ 5 are below 90 for these two scenes (the error for the not estimated images are 180 degrees for every neighbor). For kicker, however, all images are correctly registered.

[lpanaf]

@w-hc As for the command for creating the database, what I did is that I extract features with colmap feature_extractor directly, but I manually create new cameras with the ground truth intrinsics in the database and change the associated camera ids to the newly created camera. In this case, I can handle the case where more than 1 ground truth intrinsics are provided (for example, courtyard)

[mattiadurso]

but I manually create new cameras with the ground truth intrinsics in the database and change the associated camera ids to the newly created camera.

I also did this, but with the gui

I get for scene (all images, registered)

• electro (45,43)
• kicker (31,31)
• lounge (10,6)

[lpanaf]

but I manually create new cameras with the ground truth intrinsics in the database and change the associated camera ids to the newly created camera.

I also did this, but with the gui

I get for scene (all images, registered)

• electro (45,43)
• kicker (31,31)
• lounge (10,6)

Thanks @mattiadurso for replying. I got exactly the same number as this :)

[w-hc]

Thanks I'm able to roughly match the numbers.

[w-hc]

@lpanaf except that on 'meadow', glomap with gt intrinsics completely fails for some reason. The AUC is basically 0. I am using the latest commit. I notice the norms of the pose locations are close to 0.
I also ran without gt intrinsics, and for meadow I can get 55.4 AUC@1 and 82.6 AUC@3.