Clarify alignment of memory provided to applications

[devreal]

Problem

The MPI standard does not mandate any alignment requirements for functions that allocate memory that is provided to the user, which include MPI_Alloc_mem, MPI_Win_allocate, and MPI_Win_allocate_shared. Misaligned memory can lead to unexpected errors as the compiler is free to issue instructions that require a specific minimum alignment specific to the machine, e.g., vmovaps to handle long double types on 64-bit systems, which requires 16B alignment.

In contrast to MPI, POSIX memory allocation functions (malloc etc) guarantee memory alignment to be suitable for any built-in type.

Proposal

This proposal clarifies the alignment requirements for MPI_Alloc_mem to be suitable for load/store accesses of any pre-defined MPI type: The allocated memory shall be byte-aligned to at least the
alignment required for load/store accesses of any datatype corresponding to a predefined
MPI datatype. It also adds an info key minimum_memory_alignment that allows users to request a specific alignment, similar to posix_memalign. Both additions apply to MPI_Win_allocate[_shared] through the back-reference to MPI_Alloc_mem in the corresponding sections. For contiguous shared memory allocated through MPI_Win_allocate_shared, the alignment requirements only apply to the first process with non-zero size argument.

Changes to the Text

Changes detailed in PR: https://github.com/mpi-forum/mpi-standard/pull/96

Changes highlighted in the standard PDF with ticket121:
mpi-report.pdf

Updated annotated pdf (only ticket-0 and no-no-vote changes since
first pdf on two-week-deadline, Feb. 18, 2019), changes highlighted with ticket121:
mpi-report_issue-121_2019-02-21_21.35-1.pdf

Older version from Feb 18, 2019, changes highlighted with ticket121:
mpi-report.pdf

Updated annotated pdf as of May 10, 2019. Changes highlighted with ticket121:
mpi-report_issue-121_2019-05-10_19.16-1.pdf

Updated annotated pdf as of May 14, 2019 for the reading in Chicago. Changes highlighted with ticket121:
mpi-report_issue-121_2019-05-14_11.22.pdf

Updated annotated pdf as of June 25, 2019 for the virtual meeting on July 3. Changes highlighted with ticket121:
mpi-report-issue-121_2019-06-25_11.32.pdf

Updated annotated pdf as of August 22, 2019 for the reading in Zürich. Changes highlighted with ticket121:
mpi-report_issue-121_2019-08-22_09.28.pdf

Updated annotated pdf as of November 11, 2019 for the reading in Albuquerque. Changes highlighted with ticket121:
mpi-report-issue-121_2019-11-25-13.28.pdf

Impact on Implementations

Implementations have to ensure proper alignment of memory provided to the user, which may require proper alignment of internal meta-data. Support for the new info key may require the use of special allocation functions (e.g., posix_memalign) or "manually" inserting additional padding.

Impact on Users

Users can rely on proper alignment of provided memory to avoid unexpected errors and may request specific alignments specify desired alternative minimum alignment to enable higher performance on certain platforms/configurations, e.g., by providing alignment hints to a vectorizing compiler. There is no negative impact on users.

References

Issue with window memory alignment in Open MPI when using long double in a window: open-mpi/ompi#4952

A discussion in the RMA-WG: mpiwg-rma/rma-issues#3

Other libraries allocating memory handed out to the application have the following conventions:

• POSIX malloc:

  The pointer returned if the allocation succeeds shall be suitably aligned so that it may be assigned to a pointer to any type of object and then used to access such an object in the space allocated

• CUDA cudaMalloc:

  The allocated memory is suitably aligned for any kind of variable.

• OpenMP 5.0 allocate, §2.11.4:

  The allocation of each list item will be byte aligned to at least the alignment required by the base language for the type of that list item.

[jeffhammond]

We should also have a standardized alignment info key, because there are good reasons to request alignment of 64B, 128B, 4KiB, 2MiB and 1GiB, among others.

[devreal]

Updated annotated pdf (only ticket-0 and no-no-vote changes since
first pdf on two-week-deadline, Feb. 18, 2019), changes highlighted with ticket121:
mpi-report_issue-121_2019-02-21_21.35-1.pdf

Older version from Feb 18, 2019, changes highlighted with ticket121:
mpi-report.pdf

[devreal]

Updated annotated pdf, changes highlighted with ticket121:
mpi-report_issue-121_2019-05-10_19.16-1.pdf

[devreal]

Version of annotated proposal for reading in Chicago:
mpi-report_issue-121_2019-05-14_11.22.pdf

This version adds an AtoI that implementations may rely the underlying system memory allocation routines to provide properly aligned memory. The PR has been updated accordingly.

[devreal]

Updated proposal based on the discussion in Chicago and points raised in the PR:
mpi-report-issue-121_2019-06-25_11.32.pdf

This version will be used in the virtual meeting reading on July 3.

Changes include the definition of default alignment and the info key only in the MPI_Alloc_mem section and references to it in the window allocation sections plus a more detailed discussion of the issues surrounding memory alignment in contiguous shared memory windows.

[devreal]

Updated proposal based on the feedback at the virtual meeting:

mpi-report_issue-121_2019-08-21_10.33.pdf

[devreal]

Updated proposal based on the virtual meeting and comments from @dholmes-epcc-ed-ac-uk:

mpi-report_issue-121_2019-08-22_09.28.pdf

[devreal]

Proposal for the first vote in Albuquerque (changes marked with ticket121):

mpi-report-issue-121_2019-11-25-13.28.pdf

The ticket-0 changes before the first vote highlighted in the following document (again using ticket121):

mpi-report-ticket0-dec2019.pdf

[wesbland]

This passed the no-no and first vote in Albuquerque, New Mexico on 2019-12-12.

[wesbland]

This proposal passed a second and final vote at the February 2020 meeting:

Yes - 24
No - 0
Abstain - 0