From 4f463a945a0675e9ea072d1537ed9a94ffe41245 Mon Sep 17 00:00:00 2001 From: "Eric T. Johnson" Date: Mon, 19 Feb 2024 20:38:14 -0500 Subject: [PATCH 1/2] Add note about inlining on Frontier --- sphinx_docs/source/olcf-compilers.rst | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/sphinx_docs/source/olcf-compilers.rst b/sphinx_docs/source/olcf-compilers.rst index 5373563..403ee7f 100644 --- a/sphinx_docs/source/olcf-compilers.rst +++ b/sphinx_docs/source/olcf-compilers.rst @@ -85,3 +85,22 @@ build via: .. prompt:: bash make COMP=gnu USE_HIP=TRUE + + +HIP Function Inlining +^^^^^^^^^^^^^^^^^^^^^ + +By default, the ROCm compiler inlines all function calls in device code +(for better compatibility with codes that use file- or function-scoped +``__shared__`` variables). This greatly increases the time it takes to +compile and link, and may be detrimental for the templated Microphysics +networks with lots of compile-time loop unrolling. + +This can be disabled by passing flags to ``hipcc`` to allow non-inlined +function calls: + +.. prompt:: bash + + make USE_HIP=TRUE EXTRACXXFLAGS='-mllvm -amdgpu-function-calls=true' + +See also https://rocm.docs.amd.com/en/docs-5.3.3/reference/rocmcc/rocmcc.html#rocm-compiler-interfaces From c43238efd26bec2f1d5c0a67d5c82a6850cf15ee Mon Sep 17 00:00:00 2001 From: "Eric T. Johnson" Date: Mon, 19 Feb 2024 20:38:58 -0500 Subject: [PATCH 2/2] USE_HIP=TRUE overrides COMP=gnu See amrex/Tools/GNUMake/Make.defs --- sphinx_docs/source/olcf-compilers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sphinx_docs/source/olcf-compilers.rst b/sphinx_docs/source/olcf-compilers.rst index 403ee7f..a532629 100644 --- a/sphinx_docs/source/olcf-compilers.rst +++ b/sphinx_docs/source/olcf-compilers.rst @@ -84,7 +84,7 @@ build via: .. prompt:: bash - make COMP=gnu USE_HIP=TRUE + make USE_HIP=TRUE HIP Function Inlining