Performance Analysis

You are here: Home > Developer Documentation > Performance Analysis

---

This page presents information on using performance analysis tools with PIConGPU.

Score-P

Update early 2016 (Score-P 1.X): Michael Sippel's Gist

Score-P is a measurement infrastructure combining several open-source performance analysis tools. It enables to trace and profile massively-parallel applications, including hybrid MPI+CUDA programs.

PIConGPU has cmake support for Score-P. When building and installing the measurement tool, be sure to enable support for MPI and CUDA (and CUPTI).

<user>:<scorep-build-dir>$ ./configure ... --enable-mpi --enable-cuda

When configuring PIConGPU, use the Score-P wrapper scripts for the C++ and NVCC compiler:

# Switch off instrumentation by setting SCOREP_WRAPPER=OFF
<user>:<pic-build-dir>$ SCOREP_WRAPPER=OFF $PICSRC/configure -a sm_35 \
                            -c "-DCMAKE_CXX_COMPILER=`which scorep-CC` \
                            -DCUDA_NVCC_EXECUTABLE=`which scorep-nvcc`" \
                            $INST_PATH/paramSets/case003-scorep

# Set instrumentation flags, --user is optional
<user>:<pic-build-dir>$ export SCOREP_WRAPPER_INSTRUMENTER_FLAGS="--cuda --mpp=mpi"
<user>:<pic-build-dir>$ make -j
<user>:<pic-build-dir>$ make install

Before executing PIConGPU, several Score-P environment variables must be set in your batch environment template script. Some template scripts already provide these environment variables, e.g. jaguar/batch_scorep_profile.tpl. For your own script, set at least the following (buffer sizes may vary):

export SCOREP_ENABLE_TRACING=yes
export SCOREP_CUDA_ENABLE=yes,flushatexit
export SCOREP_CUDA_BUFFER=200M
export SCOREP_TOTAL_MEMORY=1G
export SCOREP_FILTERING_FILE=!TBG_dstPath/tbg/scorep.filter

When successfull, a new directory called scorep-* is created which contains the trace file trace.otf2. The trace can than be visualized with Vampir.

Further information:

• Score-P website
• Online documentation
• Score-P training material slides