Use The_Comms_Arena in particle communication.

[atmyers]

The proposed changes:

• fix a bug or incorrect behavior in AMReX
• add new capabilities to AMReX
• changes answers in the test suite to more than roundoff level
• are likely to significantly affect the results of downstream AMReX users
• include documentation in the code and/or rst files, if appropriate

[WeiqunZhang]

Because The_Comms_Arena is an alias to The_Pinned_Arena when gpu aware mpi is not used, the code like below might be problematic.

    // assuming ParallelDescriptor::UseGpuAwareMpi() is false                                                                                                                

    amrex::PODVector<char, PolymorphicArenaAllocator<char> > rcv_buffer;
    rcv_buffer.setArena(The_Comms_Arena()); // pinned memory if not using gpu aware mpi                                                                                      

    if (ParallelDescriptor::UseGpuAwareMpi())
    {
        // ...                                                                                                                                                               
    }
    else
    {
        ...
        // rcv_buffer is not device memory                                                                                                                                   
        Gpu::htod_memcpy_async(rcv_buffer.dataPtr(), pinned_rcv_buffer.dataPtr(), pinned_rcv_buffer.size());
    }

[atmyers]

Maybe we can use The_Comms_Arena() with gpu-aware MPI, and The_Arena() without it.

[WeiqunZhang]

Maybe we can use The_Comms_Arena() with gpu-aware MPI, and The_Arena() without it.

Yes, that would require the least amount of changes.

[ax3l]

This is causing a memory overhead and performance overhead in ImpactX: ECP-WarpX/impactx#769

Let's discuss next week how we proceed (e.g., opt-in to this PR's features, changing defaults, etc.).