CMake error when running dace_gpu framework

[tongzhou80]

Hi,

I got the following CMake error when running dace_gpu with any of the benchmarks:

  File "/nethome/tzhou80/anaconda3/lib/python3.11/timeit.py", line 239, in repeat                                                                                                  [299/1944]
    return Timer(stmt, setup, timer, globals).repeat(repeat, number)                                                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                         
  File "/nethome/tzhou80/anaconda3/lib/python3.11/timeit.py", line 206, in repeat                                                                                                            
    t = self.timeit(number)                                                                                                                                                                  
        ^^^^^^^^^^^^^^^^^^^                                                                                                                                                                  
  File "/nethome/tzhou80/anaconda3/lib/python3.11/timeit.py", line 178, in timeit                                                                                                            
    timing = self.inner(it, self.timer)                                                                                                                                                      
             ^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                      
  File "<timeit-src>", line 6, in inner                                                                                                                                                      
  File "/nethome/tzhou80/projects/dace/dace/sdfg/sdfg.py", line 2214, in compile                                                                                                             
    shared_library = compiler.configure_and_compile(program_folder, sdfg.name)                                                                                                               
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                               
  File "/nethome/tzhou80/projects/dace/dace/codegen/compiler.py", line 225, in configure_and_compile                                                                                         
    raise cgx.CompilerConfigurationError('Configuration failure:\n' + ex.output)                                                                                                             
dace.codegen.exceptions.CompilerConfigurationError: Configuration failure:                                                                                                                   
-- The C compiler identification is NVHPC 23.9.0                                                                                                                            
-- The CXX compiler identification is NVHPC 23.9.0                                                                                                                          
-- Detecting C compiler ABI info                                                                                                                                            
-- Detecting C compiler ABI info - done                                                                                                                                     
-- Check for working C compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/23.9/compilers/bin/nvc - skipped                                                                          
-- Detecting C compile features                                                                                                                                             
-- Detecting C compile features - done                                                                                                                                      
-- Detecting CXX compiler ABI info                                                                                                                                          
-- Detecting CXX compiler ABI info - failed                                                                                                                                 
-- Check for working CXX compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/23.9/compilers/bin/nvc++                                                                                
-- Check for working CXX compiler: /opt/nvidia/hpc_sdk/Linux_x86_64/23.9/compilers/bin/nvc++ - broken                                                                       
CMake Error at /nethome/tzhou80/anaconda3/lib/python3.11/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeTestCXXCompiler.cmake:60 (message):                         
  The C++ compiler                                                                                                                                                          
    "/opt/nvidia/hpc_sdk/Linux_x86_64/23.9/compilers/bin/nvc++"                                                                                                             
                                                                                                                                                                            
  is not able to compile a simple test program.                                                                                                                             
                                                                                                                                                                            
  It fails with the following output:                                                                                                                                       
                                                                                                                                                                            
    Change Dir: '/nethome/tzhou80/projects/npbench/.dacecache/fusion/build/CMakeFiles/CMakeScratch/TryCompile-PAB6f7'                                                       
                                                                                                                                                                            
    Run Build Command(s): /nethome/tzhou80/anaconda3/lib/python3.11/site-packages/cmake/data/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_11b32/fast          
    /usr/bin/gmake  -f CMakeFiles/cmTC_11b32.dir/build.make CMakeFiles/cmTC_11b32.dir/build                                                                                 
    gmake[1]: Entering directory '/nethome/tzhou80/projects/npbench/.dacecache/fusion/build/CMakeFiles/CMakeScratch/TryCompile-PAB6f7'                                      
    Building CXX object CMakeFiles/cmTC_11b32.dir/testCXXCompiler.cxx.o                                                                                                     
    /opt/nvidia/hpc_sdk/Linux_x86_64/23.9/compilers/bin/nvc++   -std=c++14 -fPIC -Wall -Wextra -O3 -march=native -ffast-math -Wno-unused-parameter -Wno-unused-label  -MD -MT CMakeFiles/cmTC
_11b32.dir/testCXXCompiler.cxx.o -MF CMakeFiles/cmTC_11b32.dir/testCXXCompiler.cxx.o.d -o CMakeFiles/cmTC_11b32.dir/testCXXCompiler.cxx.o -c /nethome/tzhou80/projects/npbench/.dacecache/fus
ion/build/CMakeFiles/CMakeScratch/TryCompile-PAB6f7/testCXXCompiler.cxx                                                                                                     
    nvc++-Error-Unknown switch: -ffast-math                                                                                                                                 
    nvc++-Error-Unknown switch: -Wno-unused-label                                                                                                                           
    gmake[1]: *** [CMakeFiles/cmTC_11b32.dir/build.make:79: CMakeFiles/cmTC_11b32.dir/testCXXCompiler.cxx.o] Error 1                                                        
    gmake[1]: Leaving directory '/nethome/tzhou80/projects/npbench/.dacecache/fusion/build/CMakeFiles/CMakeScratch/TryCompile-PAB6f7' 

I think the issue is somehow the discovered C++ compiler is not able to compile a simple test program. However, when I tried to manually compile a simple test CUDA program using the same compiler /opt/nvidia/hpc_sdk/Linux_x86_64/23.9/compilers/bin/nvc++, it worked fine. /opt/nvidia/hpc_sdk/Linux_x86_64/23.9/compilers/bin/ is my CUDA toolkit directory. Any idea what could be the issue? Thanks!

[alexnick83]

    /opt/nvidia/hpc_sdk/Linux_x86_64/23.9/compilers/bin/nvc++   -std=c++14 -fPIC -Wall -Wextra -O3 -march=native -ffast-math -Wno-unused-parameter -Wno-unused-label  -MD -MT CMakeFiles/cmTC
_11b32.dir/testCXXCompiler.cxx.o -MF CMakeFiles/cmTC_11b32.dir/testCXXCompiler.cxx.o.d -o CMakeFiles/cmTC_11b32.dir/testCXXCompiler.cxx.o -c /nethome/tzhou80/projects/npbench/.dacecache/fus
ion/build/CMakeFiles/CMakeScratch/TryCompile-PAB6f7/testCXXCompiler.cxx                                                                                                     
    nvc++-Error-Unknown switch: -ffast-math                                                                                                                                 
    nvc++-Error-Unknown switch: -Wno-unused-label

It looks like a CMake/DaCe configuration issue. I believe DaCe's default C/C++ compilation flags are for the GNU compiler, and they do not work with nvcc/nvc++. DaCe will use nvcc to compile CUDA kernels but will call gcc (or the C/C++ compiler found on the system by CMake, nvc++ in your case) for host code. You can probably solve the issue either by explicitly telling CMake to use a different C/C++ compiler (CC and CXX environment variables) or by editing the DaCe configuration file (~/.dace.conf) to use nvc++-compatible flags for the host (CPU) compiler (this link may be helpful; also this).

[alexnick83]

If you want to change the flags, then the .dace.conf in your home folder should have the following form:

compiler:
  cpu:
    args: -std=c++14 <insert other nvcc/nvc++ compatible flags>

You may need to clean up the build folder (in your example above, that would be /nethome/tzhou80/projects/npbench/.dacecache/fusion) for the changes to take effect.

You can also explicitly set the host compiler in the DaCe configuration file, which should hopefully be properly passed to CMake:

compiler:
  cpu:
    executable: <path to C/C++ compiler>