Fix land albedo regridding

[LenkaNovak]

This is the output of CL.surface_albedo(sim.model, sim.integrator.u, sim.integrator.p) when using map_temporal

We checked the original nc file and this seems to be fine:

This seems to be a problem in ClimaLand, but it would be good to check that this is still the case for the latest ClimaLand tag.

[Sbozzolo]

A first look in ClimaUtilities shows no clear problems with TempestRemap:

CliMA/ClimaUtilities.jl#53

[Sbozzolo]

This is from ClimaLand, surface_albedo, at the first iteration for our CI experiment

[juliasloan25]

I tried to recreate this from ClimaCoupler using the slabplanet_albedo_temporal_map configuration. The albedo looks fine after initialization, after 2 timesteps, and after 1 hour of coupled simulation. Here's the plot after one hour:

Was this seen in only the AMIP case before? If not, maybe this was fixed by updating to the most recent release of ClimaLand (v0.12.0), which uses ClimaUtilities for regridding

[juliasloan25]

I also tried the AMIP case (using the gpu_amip_albedo_temporal_map config, but running on CPU) and the albedo still looks fine after 1 hour:

[Sbozzolo]

If you have the scripts, can you also try with the TempestRegridder? I think the change in ClimaLand 0.12 was moving to the InterpolationsRegridder

[juliasloan25]

I tried using ClimaLand v0.12.1 and TempestRegridder. This what the albedo looks like after initialization, using the slabplanet_albedo_temporal_map config with the resolution increased to 16 horizontal elements.

I also see basically the same plot when using the AMIP setup instead slabplanet. It's very similar to the original picture Lenka showed, so it seems this is still something we need to investigate

[juliasloan25]

I also took a look at the result of ClimaLand.surface_albedo in the ClimaLand temporal map experiment, and I see this after initialization:

and this after 50 days of simulation time:

[Sbozzolo]

Is your conclusion that it works in Land but not in the Coupler? Or are the white boxes a concern?

[juliasloan25]

Is your conclusion that it works in Land but not in the Coupler? Or are the white boxes a concern?

I don't know what conclusion to draw yet. I'm a bit concerned about the white boxes, and I'm also confused why these plots have them but the ones you sent earlier don't

[Sbozzolo]

I went back to ClimaLand 0.11, pre-ClimaUtilities, and found that the issue was already there:

To reproduce, change ClimaLand.jl/src/shared_utilities/Domains.jl

- horztopology = ClimaCore.Topologies.Topology2D(comms_ctx, horzmesh)
+ horztopology = ClimaCore.Topologies.Topology2D(comms_ctx, horzmesh, ClimaCore.Topologies.spacefillingcurve(horzmesh))

Run

using ClimaLand, ClimaCoreTempestRemap, Plots, ClimaCorePlots
domain = ClimaLand.Domains.SphericalShell(radius=6e6, depth=1., nelements=(50, 10), npolynomial=3)
albedo = ClimaLand.Bucket.PrescribedBaregroundAlbedo{Float64}(1.0, domain.space.surface)
plot(albedo.α_bareground)

(cc: @sriharshakandala)

The issue is probably with ClimaUtilities.TempestRemapper (which inherited the land remapper) given that the ClimaCoupler is using ClimaCoreTempestRemap with spacefilledcurves, right?

[juliasloan25]

comment copied from #818:

The problem is in boundary_space somehow. I created a horzspace from scratch from the coupler and it works. What I noticed is that the boundary_space mesh has a different elemorder compared to a fresh horzspace

julia> boundary_space.grid.topology.elemorder
2400-element Vector{CartesianIndex{3}}:
 CartesianIndex(1, 1, 1)
.....

julia> horzspace.grid.topology.elemorder
CartesianIndices((20, 20, 6))

These are not equivalent.

Interestingly,

julia> extrema(parent(good) .- parent(bad))
(0.0, 0.0)

where good and bad are albedo obtained using horzspace or boundary_space respectively.

ClimaAtmos and ClimaCoupler both create spaces using a spacefillingcurve (e.g. here in ClimaCoupler.Regridder), whereas ClimaLand and ClimaUtilities both create spaces without a spacefillingcurve (e.g. here in ClimaUtilities.TempestRegridder). This is why the two spaces have different element orders.

I had to add a TempestRemap GPU workaround in ClimaLand to force the regridding to happen on the CPU (this involves creating a new CPU context and space), and I noticed that if I created the undistributed space with a spacefillingcurve, space_distributed == space_undistributed was not true. This is why ClimaLand (and ClimaUtilities) don't use the spacefillingcurve. We should unify this across all repos, but I'm not sure which approach we should take

cc @sriharshakandala

[juliasloan25]

Using the fix implemented in CliMA/ClimaUtilities.jl#66, we can regrid the land albedo using TempestRegridder and get the following regridded field:

There are still some things that aren't ideal in this implementation (i.e. we still need swap_space) but it seems reliable for now. We will continue using InterpolationsRegridder as the default, but this fix will let us use TempestRemap if we want to.

The other fields that are currently regridded using TempestRemap (SST, SIC, CO2, land_area_fraction) will be investigated as part of #860