Enabling GPUS in CIME

[jedwards4b]

I have created a cesm branch in my fork called gpu_improvements:

git clone https://github.com/jedwards4b/CESM.git -b gpu_improvements
cd cesm./manage_externals/checkout_externals

There are two new options to create_newcase --gpu-type and --gpu-offload
I have reworked the tests that you defined in cam to use smaller resolutions and also the se dycore and different numbers of gpus per node.The compiler is nvhpc not nvhpc-gpu - compiler definitions with -gpu will be removed by the time this PR is accepted.   I am also going to share this
with the e3sm team for feedback prior to merging so there may be further changes. 

One question I have is what to do if there is more than one offload method in the model, one section using OpenACC and another using OpenMP?I'm not sure how we would handle that.

[jedwards4b]

See changes in cime here: #4334

[jedwards4b]

Changes in other components are here:
ESMCI/ccs_config_cesm#66
ESCOMP/CMEPS#318
ESCOMP/CAM#682

[rljacob]

Can you provide some create_newcase useage examples? Or a machine file example?

[bartgol]

When running crete_newcase (rather than create_test), does one have to manually specify the gpu type? Can't it be retrieved from machine specs?

[jedwards4b]

we have machines with multiple nodes with different gpu types and multiple compilers not all of which support gpus. So no I don't think we can do that.

[bartgol]

Can it at least have a "default" value? Some machines have a single gpu type, so it can be handy to have it hard coded in config_machines.xml, and save the user a few key strokes (when possible).

[jedwards4b]

Another case is that a user wants to use the given compiler and machine but does not want to use gpus. I'll keep this suggestion in mind, but I don't know of a way to do it at the moment.

[bartgol]

In that case, I would expect the user to not pass --gpu-offload, which would make CIME ignore any other GPU setting.

Anyhow, this is just a super minor thing.

[sjsprecious]

Thanks @jedwards4b for working on this! Regarding your question, I am not sure whether it is worth the effort to consider the case that both OpenACC and OpenMP offload are used at the same time. If you do want to account for this type of situation, we can apply the compilation flags -acc -gpu=cc70 -mp=gpu for the NVHPC compiler, which would compile both OpenACC and OpenMP offload codes at the same time. However, in my opinion, by doing so, we probably do not need the gpu_offload variable anymore.

[jedwards4b]

So we could define an offload method 'combined' which would support both.

[sjsprecious]

That sounds like a good option!

[jedwards4b]

From @sarats : As this mandates GPU_OFFLOAD, we probably have to add a third option to handle the case for C++ codebases that use a library approach as it won't be OpenMP or OpenACC.

[sarats]

The combined mode would also help C++ components built with Fortran+Directives componenents in a coupled configuration. But one has to distinguish between OpenACC and OpenMP within the "combined" conditionals if needed to define flags. There will always be configuration complexity (with or without GPU_OFFLOAD var) to handle special cases, so it may not lead to concision in those cases.

[sarats]

Can you illustrate an example of where specialization for GPU_TYPE resides? In CIME machine section?

[jedwards4b]

I'm not sure if I understand the question but for cesm we have cmake flags in: https://github.com/ESMCI/ccs_config_cesm/pull/66/files#diff-e9435a30e3faf12b3429137c9ac98de1a71394ad1c3dc17f284af32d9f8cf4b8R52

[oksanaguba]

From e3sm perspective, is it needed in e3sm? right now we have cmake macros files for "compiler"/machine pairs (where "compiler" can be any config we want to support) with flexible cmake vars, including conditional ones for each component. So, would this PR add functionality?

[jedwards4b]

Again that is what I am trying to undo - we don't need to completely duplicate compiler definitions based on whether or not we want to use gpus.

[bartgol]

I agree. Although, if we came up with a way to "inherit" XML nodes, we could have

<machine MACH="summit-base">
  all common settings
</machine>

<machine MACH="summit-gpu" inherit="summit-base">
  gpu specific settings
</machine>

This would not be completely flexible, for it would still have duplication in case 2+ gpu archs need to be used at the same time (I wasn't aware this was ever the case, btw, I always thought one GPU arch at a time).

[bartgol]

So, tbc, I agree with oksana that we can already get this behavior by multiplying mach files. But I agree that we should split the threading model concept from the machine/compiler specs.

[rljacob]

What we decided in E3SM: for machines like perlmutter which provide one front end to two different node types (cpu-only nodes and cpu-gpu nodes), we have two machine entries. For machines like Summit where there is only one type of heterogeneous node, we have one machine entry but two compiler entries: one for cpu-only compile and one for gpu compile. As long as we can keep doing that, this won't impact E3SM.

[rljacob]

I see how the one-machine-entry-per-node-type can get very duplicative when you have N>4 node types hanging off a single front end.

[bartgol]

I wonder if GPU and CPU threading can be unified in a single option. Something like --threading=XYZ, with possible XYZ values none, openmp, openmp-target, openacc, cuda, hip,... Maybe even a list of those, to enable, say, OpenMP on host, and both openacc and cuda on device (if some components use the former while others use the latter).

[sjsprecious]

And one more question: does E3SM use OpenACC or OpenMP offload for GPU? I thought E3SM used KoKKos for GPU offload so this change may not be used in E3SM at all? Or maybe we should add kokkos as a valid gpu_offload value?

[jedwards4b]

@sjsprecious I think that it would be cuda not kokkos, but e3sm uses a different mechanism so I don't think we need to worry about that yet.

[bartgol]

Well, E3SM can currently use several ways. Kokkos is one. YAKL is a similar (smaller) library. But other components still use openacc, I think. Either way, I think E3SM is moving toward a more cmake-target-centric approach (at least I hope it will). But I would still be interested in grabbing threading model specs from the CIME case configuration (rather than rely on simply machine files). I think, at least for the ATM, what we would need to make our build decision is knowing what kind of device we're using. After that, we can configure things internally via CMake.

[sjsprecious]

Thanks @jedwards4b and @bartgol for your clarifications. That helps a lot! Sounds like the ATM component in E3SM can fully run on GPU now?

[bartgol]

Not in current release of E3SM. But yes, we have a version of ATM for ultra high res that is fully running on GPU end to end (except I/O, obviously).