distortion loss impl. #244
Conversation
|
@liruilong940607, I think there are some version issues with the Nerfacc library. I have Nerfacc Version: 0.5.3 installed (with The error a user gets: This can be fixed by installing from the git source but just a heads up... |
| ) | ||
| torch.testing.assert_close(render_colors, _render_colors) | ||
| torch.testing.assert_close(render_alphas, _render_alphas) | ||
| torch.testing.assert_close(render_distloss, _render_distloss) | ||
|
|
||
| # backward | ||
| v_render_colors = torch.randn_like(render_colors) |
There was a problem hiding this comment.
Not an issue, just a general question regarding the use of torch.autograd.grad(outputs * v_outputs ,inputs) in the gradient tests. Why do we scale outputs by these random v_outputs weights? Shouldn't torch.autograd.grad(outputs, inputs) suffice for gradient checking?
There was a problem hiding this comment.
torch.autograd.grad needs to take in a scalar in the first argument, which is essentially the loss. So here I'm testing against this formula:
dL_d(input) = dL_d(output) * d(output)/d(input)
Since we don't have the loss formula, I'll using a random tensor (v_output) to mimic dL_d(output), so the "loss" can be set to be loss = (output * v_output).sum().
Then the testing of dL_d(input) is:
torch.autograd.grad((outputs * v_outputs).sum() ,inputs)
There was a problem hiding this comment.
Makes sense, thanks. Here is a small visual of the dist loss as a heatmap whilst training:
There was a problem hiding this comment.
Very cool color theme! What theme are you using here?
The reason I haven't merge this PR is that I didn't find this distortion loss to be helpful on ANY of the scenes in MipNeRF 360, which is a bit frustrated and not expected.
Let me know if you find anything otherwise!
There was a problem hiding this comment.
I'm guessing the reason that this distortion loss isn't useful might have something to do with the imperfect depth accumulation formulation that we are currently using. The depth formulation from Gaussian Opacity Fields or RaDe-GS seems to be more accurate, which might be a prerequisite for distortion loss to work.
Or, maybe its just that there aren't many floaters in these GS scenes anyway?
There was a problem hiding this comment.
For those like me who were wondering what the awesome color scheme was, I think I tracked it down to cmr.voltage. https://cmasher.readthedocs.io/user/sequential/voltage.html
|
I tried this distortion loss with 2DGS and below are the results: (without distortion loss) Seems like the improvement is very subtle and focus on details; and this is also the case for the original 2DGS implementation; I will try the L2 version in the 2DGS paper next |
|
@FantasticOven2 I think the speckle artifacts you have in your normal image are from overflowing uint8 in the visualization. You may need to clamp or normalize your normals. |
|
Hi Ruilong, do you have any plans to merge this? I think it would be helpful in some more underconstrained settings. |
I haven't got a case where I find it useful and that's why I wasn't quite motivated to spend further efforts on merging it. But I guess it's true that all the cases I tried are well constrained. I could try to find some cycles to merge this but no guarantee on the timeline! |
Implement distortion loss for 3DGS from Mip-NeRF 360:
Note: Since the distortion loss is a per-pixel value, I implement it as a per-pixel map rendered from the
rasterization()function, with the same resolution with the rendered image. This could be reduced into a scalar with a.mean()call on it for backward. Having the distortion loss as a map instead of a scalar could be useful for visualization and analysis purposes.