MaterialGrid in parallel MEEP - ram consumption

[emoryljenkins]

Hi guys,

I am running a simulation with a rather complex geometry (see black silicon) so I have predefined the structure as a MaterialGrid. It's a pretty fine structure so I need high resolution and it's random so I need the simulation domain to be fairly large before thinking about tiling. This means the MaterialGrid is measured in gigabytes.

I am running into ram consumption issues since it appears to me that each thread opened up by MPI has its own copy of the entire MaterialGrid. Therefore, ram usage increases quite a bit with increasing core count.

Is there a way to mitigate this? Ideally each process should only have in its memory the portion of the MaterialGrid corresponding to its chunk. I'm also open to suggestions for other ways to define an arbitrary structure that don't create this problem.

Thanks,
Emory

[stevengj]

Right now, material grids can't be distributed — every process gets its own copy. You can mitigate it somewhat by using threads rather than MPI, though Meep's multi-threaded scaling is currently not great.

Could you define the structure by a material_function? (With a material_function you can do whatever you want, including interpolating from a per-process grid.)