2DGS #208

FantasticOven2 · 2024-06-08T07:05:20Z

This is an attempt to integrate 2DGS https://surfsplatting.github.io/ into gsplat. You can run simple 2D traning example with:
2D verification examples:
python examples/2d_example.py --model-type {3dgs, 2dgs}
Image fitting:
python examples/image_fitting.py --model-type {3dgs, 2dgs}

3D scene reconstruction:
python examples/simple_trained_2dgs.py --data-dir ...

traj_29999.1.1.mp4

Evaluations (Average over 7 scenes in Mipnerf-360 dataset)
w/o regularization

Scene	PSNR	SSIM	LPIPS	Memory (GB)	Train Time (min:sec)	Num GS
gsplat	28.76	0.8669	0.1450	3.70	15:44	1.90M
gsplat (packed)	28.74	0.8617	0.1448	3.48	15:25	1.89M
inria	28.73	0.8601	0.1480	3.73	22:16	1.93M

w/ normal consistency regularization

Scene	PSNR	SSIM	LPIPS	Memory (GB)	Train Time (min:sec)	Num GS
gsplat	27.85	0.8436	0.1853	3.78	16:48	1.96M
inria	28.06	0.8492	0.1694	3.66	22:18	1.91M

w/ normal consistency, distortion regularization

Scene	PSNR	SSIM	LPIPS	Memory (GB)	Train Time (min:sec)	Num GS
gsplat	27.80	0.8422	0.1866	3.76	16:44	1.96M
inria	28.05	0.8489	0.1697	3.61	22:06	1.88M

vye16

This is a wonderful start! I think it would be great to have similar numerical testing the same way that we have for the 3dgs, but we can coordinate this.

I'm inclined to separate the two into separate functions and interfaces, what are your thoughts @liruilong940607? There is also a lot of repeated code but that's not a big problem for now, we can figure out if/how we can share between 2dgs and 3dgs later once.

vye16 · 2024-06-10T06:43:51Z

gsplat/cuda/csrc/rasterization.cu

+    // moving the channel padding from python to C side.
+    switch (channels) {
+    case 1:
+        rasterize_to_pixels_fwd_kernel<1><<<blocks, threads, 0, stream>>>(


rasterize_to_pixels_fwd_2dgs_kernel right?

oops! Good catch!

vye16 · 2024-06-10T06:45:08Z

gsplat/cuda_legacy/_wrapper.py

@@ -300,6 +293,10 @@ def rasterize_gaussians(
    block_width: int,
    background: Optional[Float[Tensor, "channels"]] = None,
    return_alpha: Optional[bool] = False,
+    model_type: Literal["3dgs", "2dgs"] = "3dgs",


I think it's better to just make a separate function for 2dgs, but @liruilong940607 what do you think?

yes that would be my preference as well. seems like 2D GS projection and 3D GS have different argument list, though some are shared, it would be a bit confusing to squeeze all arguments into the same function.

vye16 · 2024-06-10T06:46:12Z

gsplat/cuda_legacy/_wrapper.py

@@ -548,6 +774,7 @@ def project_gaussians(
    img_width: int,
    block_width: int,
    clip_thresh: float = 0.01,
+    model_type: Literal["3dgs", "2dgs"] = "3dgs"


here again, I feel like it'd be better to have separate functions

examples/image_fitting.py

liruilong940607 · 2024-06-10T07:09:13Z

This is a wonderful start! I think it would be great to have similar numerical testing the same way that we have for the 3dgs, but we can coordinate this.

I'm inclined to separate the two into separate functions and interfaces, what are your thoughts @liruilong940607? There is also a lot of repeated code but that's not a big problem for now, we can figure out if/how we can share between 2dgs and 3dgs later once.

Yes totally agree. I definitely recommend start with developing as separate functions (or even separate files). This would make the development so much easier and faster without the need of considering compatibility and it won't interrupting what's already in there.

I imagine some helper functions might be able to get reused, like on the python side, there is sorting related functions, and maybe also rasterization? The projection is definitely different between 2D and 3D GS so would be nice to have different interfaces.

Merging is kinda like a cleanup, which we can certainly do it later.

vye16 · 2024-06-11T05:15:22Z

gsplat/cuda/csrc/projection.cu

+    glm::mat3 RS_camera = R * RS;
+    glm::mat3 WH = glm::mat3(RS_camera[0], RS_camera[1], mean_c);
+
+    glm::mat3 inverse_intrinsic = glm::mat3(


hmmm, the ellipse aabb in the original paper uses the projection matrix which transforms into clipping space, can we do this and directly take the same computation for ellipse aabb? @FantasticOven2 @liruilong940607 am I misunderstanding this?

I think using ray-splat intersection to compute AABB is more accurate than using the perspective transformation for ellipse aabb in the original paper as mentioned here; the author also worked out the math here

gchhablani · 2024-06-12T19:33:59Z

I am highly interested in having this handy.
Just curious, will this also be able to support things like Gaussian Surfels (which is pretty much the same as 2DGS), but interesting losses: https://github.com/turandai/gaussian_surfels

FantasticOven2 · 2024-06-15T14:44:11Z

Forward pass works on simple 2D experiments for v1.0

FantasticOven2 · 2024-06-15T15:09:39Z

Slant views

(3DGS)

(2DGS)

hardikdava · 2024-09-08T17:45:59Z

@FantasticOven2 I have tried to install latest version and it seems working after decreasing MAX_JOBS in setup.py. If I set to MAX_JOBS=10 then the system is crashing for me.

c.c. @liruilong940607

liruilong940607 · 2024-09-10T18:26:40Z

Oh I think the renders folder are accidentally added into the commit, because of the removal (by me) of the renders in .gitignore.

Could you remove that from this PR? Feels like maybe the results folder is enough for dumping things locally.

FantasticOven2 · 2024-09-11T18:20:00Z

Oh I think the renders folder are accidentally added into the commit, because of the removal (by me) of the renders in .gitignore.

Could you remove that from this PR? Feels like maybe the results folder is enough for dumping things locally.

Done!

liruilong940607 · 2024-09-11T22:03:28Z

formatting as one last thing!! Thanks!

FantasticOven2 · 2024-09-12T01:25:45Z

formatting as one last thing!! Thanks!

Done!

liruilong940607 · 2024-09-12T03:21:31Z

Format check is still failing. Could you run?

pip install black==22.3.0
black . gsplat/ tests/ examples/ profiling/

Also, the renders folder are submitted to this PR again ... We should probably change the path of this to results in the 2d image fitting script

FantasticOven2 · 2024-09-12T03:51:20Z

Format check is still failing. Could you run?
pip install black==22.3.0
black . gsplat/ tests/ examples/ profiling/
Also, the renders folder are submitted to this PR again ... We should probably change the path of this to results in the 2d image fitting script

the format is due to the black version mismatch; for the renders folder, do you mean examples/renders? I removed it and also change to "results" in image_fitting.py

liruilong940607

LGTM! Thanks for the long going efforts!

Master-cai · 2024-09-12T05:49:54Z

Thanks for the hard work! I want to ask can we replace the 3dgs to the 2dgs by simply replacing the rasterization() to rasterization_2dgs() for rendering?

btw, is there a bug or typo?

gsplat/examples/simple_trainer_2dgs.py

Line 454 in 2d6a967

render_colors, render_alphas = renders

It seems that no vars named renders in rasterize_splats().

Thanks!

cc @liruilong940607 @FantasticOven2

aW4KeNiNG · 2024-09-12T16:37:41Z

Thanks for the effort! Just a note, there is a missing option added in simple_trainer recently: "normalize_world_space"

FantasticOven2 · 2024-09-12T17:22:36Z

Thanks for the hard work! I want to ask can we replace the 3dgs to the 2dgs by simply replacing the rasterization() to rasterization_2dgs() for rendering?

btw, is there a bug or typo?

gsplat/examples/simple_trainer_2dgs.py

Line 454 in 2d6a967

render_colors, render_alphas = renders

It seems that no vars named renders in rasterize_splats().
Thanks!

cc @liruilong940607 @FantasticOven2

Hey, thanks for pointing this out; yes this is a typo. I will update this later on. Since inria implementation is mainly used for benchmark, it shouldn't affect the gsplat implementation.

There are a few differences between rasterization and rasterization_2dgs, especially for the returned tensors. But it should be straightforward to switch from one to the other, and you can check the doc here for reference

ozgunthirddimension · 2024-09-16T10:30:56Z

Hello, Great work again. as I was running the code I have seen that when you set the absgrad to True it does not function and throw an error that there is no absgrad as a parameter for the densify. When I check the code it still only does it for means2d and I wonder if you plan to implement it also for the densift. Thanks. Have a nice day

FantasticOven2 · 2024-09-19T03:11:12Z

Hello, Great work again. as I was running the code I have seen that when you set the absgrad to True it does not function and throw an error that there is no absgrad as a parameter for the densify. When I check the code it still only does it for means2d and I wonder if you plan to implement it also for the densift. Thanks. Have a nice day

Hey!
Thanks for your interest and sorry for the late response. The current version of 2DGS doesn't support absgrad yet. I'm thinking about an updates for additional features such as (absgrad / anti-aliased) but not sure how long will it take; on the other hand you may try to implement absgrad yourself and it should very straightforward.

sephyli · 2024-09-27T08:49:52Z

Hi @FantasticOven2, thanks for the great work!
I noticed that in 2DGS, the gradient passed to the camera extrinsics is not available for the camera rotation. Is that normal?

FantasticOven2 · 2024-09-28T04:19:04Z

Hi @FantasticOven2, thanks for the great work! I noticed that in 2DGS, the gradient passed to the camera extrinsics is not available for the camera rotation. Is that normal?

Hey, Thanks for your interest! The current 2DGS doesn't support camera pose gradient yet.

We are working on additional features for 2DGS including camera pose gradient and will make an update in the next few month.

bbartlett-nv · 2024-09-30T14:11:25Z

Great work ! super interested in using for mesh extraction; any way to use the original "render.py" from the 2DGS repo from the pytorch checkpoint generated by this gsplat implementation? Or, plans to provide mesh extraction script here?

Thanks!

FantasticOven2 · 2024-10-02T23:59:29Z

Great work ! super interested in using for mesh extraction; any way to use the original "render.py" from the 2DGS repo from the pytorch checkpoint generated by this gsplat implementation? Or, plans to provide mesh extraction script here?

Thanks!

Hey! Thanks for your interest! Yes we do plan to add mesh extraction in the next one or two months. Meanwhile you can check Gaussian Opacity Field pull request as they have a draft mesh extraction

bbartlett-nv · 2024-10-05T23:06:29Z

great news! yep I have run that PR - works pretty well but need textures. looking forward to trying out your implementation.

FantasticOven2 changed the title ~~2DGS Old Version~~ 2DGS Jun 8, 2024

FantasticOven2 marked this pull request as draft June 8, 2024 07:05

FantasticOven2 closed this Jun 8, 2024

FantasticOven2 force-pushed the main branch from 29e18b5 to 07d9188 Compare June 8, 2024 13:21

cuda legacy with 2DGS

086dfe0

FantasticOven2 reopened this Jun 8, 2024

FantasticOven2 added 5 commits June 8, 2024 18:16

clean up changes

906a955

add torch implementation

b288462

adding _wrapper for cuda legacy

436b424

working on forward (projection)

9a549d4

working on forward (rasterization)

5764be5

vye16 reviewed Jun 10, 2024

View reviewed changes

rasterize forward continue

cd7786f

vye16 reviewed Jun 11, 2024

View reviewed changes

FantasticOven2 and others added 4 commits June 11, 2024 07:26

starting backward (rasterization)

f408b13

stubs for 2dgs testing

8580e5e

Merge branch 'main' of https://github.com/FantasticOven2/gsplat

e596d75

ctn.rasterize (backward)

fecabc5

FantasticOven2 added 5 commits June 12, 2024 20:49

start projection backward

151ee3d

setup simple 2D experiments and debug forward

8cb5154

ctn. forward debugging

9a92cca

ctn. forward debugging

d6ead12

2d example works

449e1a4

add slant view

af66b5e

FantasticOven2 added 2 commits June 15, 2024 08:11

variable num points for 2D example

4c3717c

working on bwd

f15625b

cleanup

935cb1d

FantasticOven2 added 5 commits September 8, 2024 19:08

resolve ruilong's note

cf6b32f

rename ray_M to ray_transform

c5d1449

update doc

75de227

update doc

8f39645

adding docstring

14a097a

remove renders

9d80b19

Merge branch 'main' into main

24add4d

black formatting

3487c74

FantasticOven2 added 2 commits September 11, 2024 20:43

black format 22.3

fd219f8

remove renders folder in examples

8c4efba

liruilong940607 approved these changes Sep 12, 2024

View reviewed changes

liruilong940607 merged commit 2d6a967 into nerfstudio-project:main Sep 12, 2024
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2DGS #208

2DGS #208

FantasticOven2 commented Jun 8, 2024 •

edited

Loading

vye16 left a comment •

edited

Loading

vye16 Jun 10, 2024

FantasticOven2 Jun 10, 2024

vye16 Jun 10, 2024

liruilong940607 Jun 10, 2024

vye16 Jun 10, 2024

liruilong940607 commented Jun 10, 2024 •

edited

Loading

vye16 Jun 11, 2024 •

edited

Loading

FantasticOven2 Jun 11, 2024

gchhablani commented Jun 12, 2024

FantasticOven2 commented Jun 15, 2024

FantasticOven2 commented Jun 15, 2024 •

edited

Loading

hardikdava commented Sep 8, 2024

liruilong940607 commented Sep 10, 2024

FantasticOven2 commented Sep 11, 2024

liruilong940607 commented Sep 11, 2024

FantasticOven2 commented Sep 12, 2024

liruilong940607 commented Sep 12, 2024

FantasticOven2 commented Sep 12, 2024

liruilong940607 left a comment

Master-cai commented Sep 12, 2024

aW4KeNiNG commented Sep 12, 2024

FantasticOven2 commented Sep 12, 2024

ozgunthirddimension commented Sep 16, 2024

FantasticOven2 commented Sep 19, 2024

sephyli commented Sep 27, 2024 •

edited

Loading

FantasticOven2 commented Sep 28, 2024

bbartlett-nv commented Sep 30, 2024

FantasticOven2 commented Oct 2, 2024

bbartlett-nv commented Oct 5, 2024

2DGS #208

2DGS #208

Conversation

FantasticOven2 commented Jun 8, 2024 • edited Loading

vye16 left a comment • edited Loading

Choose a reason for hiding this comment

vye16 Jun 10, 2024

Choose a reason for hiding this comment

FantasticOven2 Jun 10, 2024

Choose a reason for hiding this comment

vye16 Jun 10, 2024

Choose a reason for hiding this comment

liruilong940607 Jun 10, 2024

Choose a reason for hiding this comment

vye16 Jun 10, 2024

Choose a reason for hiding this comment

liruilong940607 commented Jun 10, 2024 • edited Loading

vye16 Jun 11, 2024 • edited Loading

Choose a reason for hiding this comment

FantasticOven2 Jun 11, 2024

Choose a reason for hiding this comment

gchhablani commented Jun 12, 2024

FantasticOven2 commented Jun 15, 2024

FantasticOven2 commented Jun 15, 2024 • edited Loading

hardikdava commented Sep 8, 2024

liruilong940607 commented Sep 10, 2024

FantasticOven2 commented Sep 11, 2024

liruilong940607 commented Sep 11, 2024

FantasticOven2 commented Sep 12, 2024

liruilong940607 commented Sep 12, 2024

FantasticOven2 commented Sep 12, 2024

liruilong940607 left a comment

Choose a reason for hiding this comment

Master-cai commented Sep 12, 2024

aW4KeNiNG commented Sep 12, 2024

FantasticOven2 commented Sep 12, 2024

ozgunthirddimension commented Sep 16, 2024

FantasticOven2 commented Sep 19, 2024

sephyli commented Sep 27, 2024 • edited Loading

FantasticOven2 commented Sep 28, 2024

bbartlett-nv commented Sep 30, 2024

FantasticOven2 commented Oct 2, 2024

bbartlett-nv commented Oct 5, 2024

FantasticOven2 commented Jun 8, 2024 •

edited

Loading

vye16 left a comment •

edited

Loading

liruilong940607 commented Jun 10, 2024 •

edited

Loading

vye16 Jun 11, 2024 •

edited

Loading

FantasticOven2 commented Jun 15, 2024 •

edited

Loading

sephyli commented Sep 27, 2024 •

edited

Loading