Enable GPU-aware MPI by default

This turns on GPU-aware MPI by default.

On all current machines, simulations run faster with GPU-aware MPI enabled. Two technical issues that prevented this are now resolved: AMReX now has the communication arena, which does not use managed memory, and SLURM no longer uses cgroup isolation for GPU bindings by default.

Closes #2967.

Src/Base/AMReX_ParallelDescriptor.cpp

@@ -57,7 +57,7 @@ namespace amrex::ParallelDescriptor {
 #endif
 
 #ifdef AMREX_USE_GPU
-    bool use_gpu_aware_mpi = false;
+    bool use_gpu_aware_mpi = true;
 #else
     bool use_gpu_aware_mpi = false;
 #endif