Implement Fisheye-GS. #398
Conversation
examples/simple_trainer.py
Outdated
| @@ -55,6 +55,8 @@ class Config: | |||
| global_scale: float = 1.0 | |||
| # Normalize the world space | |||
| normalize_world_space: bool = True | |||
| # Camera model | |||
| camera_model: str = "pinhole" | |||
There was a problem hiding this comment.
minor note: it would be nice to use Literal["fisheye", "pinhole"] here and elsewhere!
|
Hello, thanks for this PR. Please let me ask some question:
|
examples/datasets/colmap.py
Outdated
| for i in range(0, width): | ||
| for j in range(0, height): | ||
| x = float(i) | ||
| y = float(j) | ||
| x1 = (x - cx) / fx | ||
| y1 = (y - cy) / fy | ||
| theta = np.sqrt(x1**2 + y1**2) | ||
| r = ( | ||
| 1.0 | ||
| + params[0] * theta**2 | ||
| + params[1] * theta**4 | ||
| + params[2] * theta**6 | ||
| + params[3] * theta**8 | ||
| ) | ||
| x2 = fx * x1 * r + width // 2 | ||
| y2 = fy * y1 * r + height // 2 | ||
| mapx[j, i] = x2 | ||
| mapy[j, i] = y2 |
|
@ichsan2895 Use the dataset download script to download the fisheye dataset and then run the benchmarking script. |
|
Amazing work, can't wait to try! How can we remap our own fisheye images to an ideal fisheye equidistant camera? I use a Samyang 8mm lens on APS-C, which is very similar to the ZipNeRF dataset (12mm fisheye on full-frame I think) and also circular fisheyes from 360 cameras, which need the black edges masked out. Thanks! |
|
The dataloader will do it automatically. You just need to use COLMAP to provide the fisheye distortion parameters. Support for masks right now is not great. You will need to follow the code and feed your own masks into the training process. |
This PR follows the formulation from Fisheye-GS and implements new fisheye_proj and fisheye_proj_vjp functions. This allows gsplat to train with a wider FOV and thus achieve higher metrics on the ZipNerf dataset. The images in the ZipNerf dataset need to be preprocessed by remapping them to an ideal fisheye (equidistant) camera. This PR also implements masking during training.
Results
Alameda scene with bilateral grid, fisheye, mcmc w/ 2M (CC PSNR: 23.77)
alameda_ellipse.mp4