Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Exawind (GPU) Build Failure on Perlmutter #572

Open
ajpowelsnl opened this issue Oct 9, 2023 · 2 comments
Open

Exawind (GPU) Build Failure on Perlmutter #572

ajpowelsnl opened this issue Oct 9, 2023 · 2 comments

Comments

@ajpowelsnl
Copy link
Contributor

  • A "bare metal" build on Perlmutter fails on amr-wind (below).
  • Nota bene: Trilinos-13.4.0 is being used, and is specified in packages.yaml. It is possible that this version (over a year old) is the problem. The current release is 14.4.0. In the 1+ years, there have been significant changes in Kokkos. The fact that the errors are undefined host / device code, suggests the possiblity of an outmoded Trilinos.
  • I will attempt a (spack-manger) build with the current Trilinos

Error

==> Installing amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx
==> No binary for amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx found: installing from source
==> No patches needed for amr-wind
==> amr-wind: Executing phase: 'cmake'
==> amr-wind: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    '/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/gmake-4.4.1-f2rv7vhqnyerq2ch37skbefh36avh7if/bin/make' '-j20'

23 errors found in build log:
     664    make[2]: Leaving directory '/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0c
            b6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o'
     665    [ 47%] Built target fcompare
     666    [ 47%] Building CUDA object CMakeFiles/amrwind_obj.dir/amr-wind/convection/incflo_godunov_ppm.cpp.o
     667    /opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/pscratch/sd/a/ajpowel/gpu_containe
            r_spack_man/spack-manager/spack/lib/spack/env/case-insensitive/CC -DAMREX_SPACEDIM=3 -DAMR_WIND_USE_NETCDF -I/pscratch/sd/a/ajpowel/g
            pu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjama
            w3bolfshinl6olf6ghx/spack-src -I/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1d
            cc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o -I/pscratch/sd/a/ajpowel/gpu_containe
            r_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl
            6olf6ghx/spack-src/submods/amrex/Tools/C_scripts -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exaw
            ind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-11.2.0/netcdf-c-4.7.4-xaosb7jql5cpbuztkhxtfzykdfe5js3l/include -isystem=/pscratch
            /sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphas
            e-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Base -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-man
            ager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/su
            bmods/amrex/Src/Base/Parser -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb
            12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Boundary -isystem=/pscra
            tch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multip
            hase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/AmrCore -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spa
            ck-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-
            src/submods/amrex/Src/LinearSolvers/MLMG -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-
            wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/LinearSolve
            rs/OpenBC -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b
            9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Particle -isystem=/pscratch/sd/a/ajpowel/g
            pu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjama
            w3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Extern/HYPRE -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/st
            age/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o
            /submods/amrex -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c2
            8c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/Utils -isystem=/pscratch/sd/a/ajpow
            el/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqx
            jamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/MOL -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage
            /spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMRe
            X-Hydro/Godunov -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c
            28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/BDS -isystem=/pscratch/sd/a/ajpowe
            l/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxj
            amaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/Projections -isystem=/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/include -is
            ystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exawind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-
            11.2.0/hypre-git.fc49a5ec6b31763dbf0b1f25c5e559859204aa91=develop-njcszkyrclumtmknksx2zy545sfdoygt/include -O3 -DNDEBUG --generate-co
            de=arch=compute_80,code=[compute_80,sm_80] -Xcompiler=-fPIC --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=
            esa_on_defaulted_function_ignored -Xcudafe --diag_suppress=implicit_return_from_non_void_function -maxrregcount=255 -Xcudafe --displa
            y_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info -std=c++17 -MD -MT CMakeFiles/amrwind_obj.dir/amr-win
            d/convection/incflo_godunov_ppm.cpp.o -MF CMakeFiles/amrwind_obj.dir/amr-wind/convection/incflo_godunov_ppm.cpp.o.d -x cu -rdc=true -
            c /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff
            4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/convection/incflo_godunov_ppm.cpp -o CMakeFiles/amrwind_obj.dir/amr-
            wind/convection/incflo_godunov_ppm.cpp.o
     668    [ 47%] Building CUDA object CMakeFiles/amrwind_obj.dir/amr-wind/convection/incflo_godunov_predict.cpp.o
     669    /opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/bin/nvcc -forward-unknown-to-host-compiler -ccbin=/pscratch/sd/a/ajpowel/gpu_containe
            r_spack_man/spack-manager/spack/lib/spack/env/case-insensitive/CC -DAMREX_SPACEDIM=3 -DAMR_WIND_USE_NETCDF -I/pscratch/sd/a/ajpowel/g
            pu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjama
            w3bolfshinl6olf6ghx/spack-src -I/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1d
            cc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o -I/pscratch/sd/a/ajpowel/gpu_containe
            r_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl
            6olf6ghx/spack-src/submods/amrex/Tools/C_scripts -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exaw
            ind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-11.2.0/netcdf-c-4.7.4-xaosb7jql5cpbuztkhxtfzykdfe5js3l/include -isystem=/pscratch
            /sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphas
            e-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Base -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-man
            ager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/su
            bmods/amrex/Src/Base/Parser -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb
            12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Boundary -isystem=/pscra
            tch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multip
            hase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/AmrCore -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spa
            ck-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-
            src/submods/amrex/Src/LinearSolvers/MLMG -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-
            wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/LinearSolve
            rs/OpenBC -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b
            9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Particle -isystem=/pscratch/sd/a/ajpowel/g
            pu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjama
            w3bolfshinl6olf6ghx/spack-src/submods/amrex/Src/Extern/HYPRE -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/st
            age/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o
            /submods/amrex -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c2
            8c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/Utils -isystem=/pscratch/sd/a/ajpow
            el/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqx
            jamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/MOL -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage
            /spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMRe
            X-Hydro/Godunov -isystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c
            28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/BDS -isystem=/pscratch/sd/a/ajpowe
            l/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxj
            amaw3bolfshinl6olf6ghx/spack-src/submods/AMReX-Hydro/Projections -isystem=/opt/nvidia/hpc_sdk/Linux_x86_64/22.7/cuda/11.7/include -is
            ystem=/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exawind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-
            11.2.0/hypre-git.fc49a5ec6b31763dbf0b1f25c5e559859204aa91=develop-njcszkyrclumtmknksx2zy545sfdoygt/include -O3 -DNDEBUG --generate-co
            de=arch=compute_80,code=[compute_80,sm_80] -Xcompiler=-fPIC --expt-relaxed-constexpr --expt-extended-lambda -Xcudafe --diag_suppress=
            esa_on_defaulted_function_ignored -Xcudafe --diag_suppress=implicit_return_from_non_void_function -maxrregcount=255 -Xcudafe --displa
            y_error_number --Wext-lambda-captures-this --use_fast_math --generate-line-info -std=c++17 -MD -MT CMakeFiles/amrwind_obj.dir/amr-win
            d/convection/incflo_godunov_predict.cpp.o -MF CMakeFiles/amrwind_obj.dir/amr-wind/convection/incflo_godunov_predict.cpp.o.d -x cu -rd
            c=true -c /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb0
            8cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/convection/incflo_godunov_predict.cpp -o CMakeFiles/amrwind_
            obj.dir/amr-wind/convection/incflo_godunov_predict.cpp.o
  >> 670    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(48): error: calling a __host__ function("a
            mr_wind::overset::asdf(double, double)") from a __device__ function("amr_wind::overset::populate_psi( ::amrex::MultiFab &,  ::amrex::
            MultiFab &, double)::[lambda(int, int, int) (instance 1)]::operator () const") is not allowed
     671    
  >> 672    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(48): error: identifier "amr_wind::overset:
            :asdf" is undefined in device code
     673    
  >> 674    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(291): error: calling a __host__ function("
            amr_wind::overset::alpha_discrete_flux(int, int, int, int, double, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<co
            nst double>  &, const  ::amrex::Array4<const double>  &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_f
            luxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFa
            b &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 1)]:
            :operator () const") is not allowed
     675    
  >> 676    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(291): error: identifier "amr_wind::overset
            ::alpha_discrete_flux" is undefined in device code
     677    
  >> 678    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(299): error: calling a __host__ function("
            amr_wind::overset::velocity_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const d
            ouble>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_fluxes( ::amrex::
            MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::M
            ultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 1)]::operator () con
            st") is not allowed
     679    
  >> 680    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(299): error: identifier "amr_wind::overset
            ::velocity_discrete_face" is undefined in device code
     681    
  >> 682    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(303): error: calling a __host__ function("
            amr_wind::overset::gp_rho_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const dou
            ble>  &, const  ::amrex::Array4<const double>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::popu
            late_sharpen_discrete_fluxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiF
            ab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, i
            nt, int) (instance 1)]::operator () const") is not allowed
     683    
  >> 684    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(303): error: identifier "amr_wind::overset
            ::gp_rho_discrete_face" is undefined in device code
     685    
  >> 686    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(309): error: calling a __host__ function("
            amr_wind::overset::alpha_discrete_flux(int, int, int, int, double, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<co
            nst double>  &, const  ::amrex::Array4<const double>  &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_f
            luxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFa
            b &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 2)]:
            :operator () const") is not allowed
     687    
  >> 688    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(309): error: identifier "amr_wind::overset
            ::alpha_discrete_flux" is undefined in device code
     689    
  >> 690    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(315): error: calling a __host__ function("
            amr_wind::overset::velocity_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const d
            ouble>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_fluxes( ::amrex::
            MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::M
            ultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 2)]::operator () con
            st") is not allowed
     691    
  >> 692    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(315): error: identifier "amr_wind::overset
            ::velocity_discrete_face" is undefined in device code
     693    
  >> 694    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(319): error: calling a __host__ function("
            amr_wind::overset::gp_rho_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const dou
            ble>  &, const  ::amrex::Array4<const double>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::popu
            late_sharpen_discrete_fluxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiF
            ab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, i
            nt, int) (instance 2)]::operator () const") is not allowed
     695    
  >> 696    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(319): error: identifier "amr_wind::overset
            ::gp_rho_discrete_face" is undefined in device code
     697    
  >> 698    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(325): error: calling a __host__ function("
            amr_wind::overset::alpha_discrete_flux(int, int, int, int, double, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<co
            nst double>  &, const  ::amrex::Array4<const double>  &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_f
            luxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFa
            b &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 3)]:
            :operator () const") is not allowed
     699    
  >> 700    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(325): error: identifier "amr_wind::overset
            ::alpha_discrete_flux" is undefined in device code
     701    
  >> 702    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(331): error: calling a __host__ function("
            amr_wind::overset::velocity_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const d
            ouble>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::populate_sharpen_discrete_fluxes( ::amrex::
            MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::M
            ultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, int, int) (instance 3)]::operator () con
            st") is not allowed
     703    
  >> 704    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(331): error: identifier "amr_wind::overset
            ::velocity_discrete_face" is undefined in device code
     705    
  >> 706    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(335): error: calling a __host__ function("
            amr_wind::overset::gp_rho_discrete_face(int, int, int, int, const  ::amrex::Array4<const double>  &, const  ::amrex::Array4<const dou
            ble>  &, const  ::amrex::Array4<const double>  &, double &, double &, double &)") from a __device__ function("amr_wind::overset::popu
            late_sharpen_discrete_fluxes( ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiF
            ab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &,  ::amrex::MultiFab &, double, double, double)::[lambda(int, i
            nt, int) (instance 3)]::operator () const") is not allowed
     707    
  >> 708    /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=
            multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/overset/sharpen_nalu_data.H(335): error: identifier "amr_wind::overset
            ::gp_rho_discrete_face" is undefined in device code
     709    
     710    20 errors detected in the compilation of "/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git
            .32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-src/amr-wind/incflo_advance.cpp".
  >> 711    make[2]: *** [CMakeFiles/amrwind_obj.dir/build.make:79: CMakeFiles/amrwind_obj.dir/amr-wind/incflo_advance.cpp.o] Error 1
     712    make[2]: *** Waiting for unfinished jobs....
     713    make[2]: Leaving directory '/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0c
            b6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o'
  >> 714    make[1]: *** [CMakeFiles/Makefile2:1583: CMakeFiles/amrwind_obj.dir/all] Error 2
     715    make[1]: Leaving directory '/pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0c
            b6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-ddqsi3o'
  >> 716    make: *** [Makefile:149: all] Error 2

See build log for details:
  /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/stage/spack-stage-amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx/spack-build-out.txt

==> Warning: Skipping build of exawind-git.d3c1aa4656fc3c6eccaec8c684671c82a3895172=multiphase-dgi3536dajqp7rakx4qgw4mxegb62l26 since amr-wind-git.32bb12d1dcc0cb6c28c94b9d3dfbbb08cdcbbff4=multiphase-ddqsi3oqxjamaw3bolfshinl6olf6ghx failed

...

==> nalu-wind: Successfully installed nalu-wind-git.90b42cad6ab7f1cfce82ac7b04d51cec89dae42e=multiphase-6vr3qakbxhidbkh6fstefpql3tu2frnk
  Stage: 16.72s.  Cmake: 16.27s.  Build: 17m 28.03s.  Install: 9.06s.  Post-install: 1.47s.  Total: 18m 12.42s
[+] /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exawind/perlmutter/2023-10-09/opt/linux-sles15-zen3/gcc-11.2.0/nalu-wind-git.90b42cad6ab7f1cfce82ac7b04d51cec89dae42e=multiphase-6vr3qakbxhidbkh6fstefpql3tu2frnk
==> Error: exawind-git.d3c1aa4656fc3c6eccaec8c684671c82a3895172=multiphase-dgi3536dajqp7rakx4qgw4mxegb62l26: Package was not installed
==> Updating view at /pscratch/sd/a/ajpowel/gpu_container_spack_man/spack-manager/snapshots/exawind/perlmutter/2023-10-09/.spack-env/view
==> Warning: Skipping external package: [email protected]%[email protected]+wrappers build_system=generic arch=linux-sles15-zen3/sy43dsx
==> Warning: Skipping external package: [email protected]%[email protected]~allow-unsupported-compilers~dev build_system=generic arch=linux-sles15-zen3/r2wfbez
==> Warning: Skipping external package: [email protected]%[email protected]~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-sles15-zen3/4ukgdvk
==> Error: Installation request failed.  Refer to reported errors for failing package(s).

real	89m32.385s

Reproducer

git clone --recursive spack-manager [email protected]:sandialabs/spack-manager.git
cd spack-manager
export SPACK_MANAGER=${PWD)
source start.sh && spack-start
./scripts/create-exawind-snapshot.sh
@ajpowelsnl
Copy link
Contributor Author

ajpowelsnl commented Nov 20, 2023 •
edited
Loading

Update: the containers (#573, #575) fix this problem by either 1) applying patches to pinned Spack version (HEAD detached at ee68baf254) or 2) making failing (TPL dependency) packages part of the base container.

Using a more recent version of Spack would likely fix most / all of the failures reported here.

@psakievich
Copy link
Collaborator

Thanks @ajpowelsnl. We'll try to update spack soon. It is old. We also need to update trilinos and cut a release of the stack. Hoping to get spack and a release done before the winter shutdown. Updating trilinos is something I'm not sure about. I will be working on that in the coming weeks though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@psakievich @ajpowelsnl