-
Notifications
You must be signed in to change notification settings - Fork 35
Graham's notes on CX1
This page is created for notes/potential improvements to the guidelines without disturbing Fabio's original notes.
These are not guidelines to follow specifically - please don't remove info from this page either.
Need to compile things with icc.
Download and compile python 2.7.3, install to home dir.
Needs bzip2 available:
wget http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz
tar xpzf bzip2-1.0.6.tar.gz
cd bzip2-1.0.6
make
make -f Makefile_libbz2_so
Make sure the .so and .h files are found by Python configure.
In order to install pip on CX1, you can use:
curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
python get-pip.py
cd build/pip
python setup.py build
python setup.py install
Install virtualenv (if you don't have it already):
git clone git://github.com/pypa/virtualenv.git
cd virtualenv/
python setup.py install --user
Create a virtualenv in your PyOP2 tree:
python -m virtualenv --distribute .venv
and activate it:
. .venv/bin/activate
You can now use pip to install packages in your virtualenv!
To leave the virutalenv you can use deactivate.
Installing dependencies:
pip install cython decorator pyyaml pytest codepy ply jinja2 mako
pip install https://launchpad.net/instant/1.0.x/1.0.0/+download/instant-1.0.0.tar.gz
pip install hg+https://bitbucket.org/fr710/pycparser#egg=pycparser-2.08
Used guidelines at: http://software.intel.com/en-us/articles/numpy-scipy-with-mkl/
you need to
module load opencl
Clone the PyOpenCL repo and edit setup.py lines:
IncludeDir("CL", []),
LibraryDir("CL", []),
to
IncludeDir("CL", ['/apps/opencl/2.0/usr/include']),
LibraryDir("CL", ['/apps/opencl/2.0/usr/lib64']),
Then install with
python setup.py build
python setup.py install
Installation as per the README
git clone http://git.tiker.net/trees/pycuda.git
cd pycuda
git submodule init
git submodule update
python configure.py
python setup.py build
python setup.py install
You need to build your own libboost_python and libboost_thread with the intel compiler and use them to build pycuda.
Use the codepy from the OP2 repository, with the CX1 branch. Some of the hardcoded boost paths will need editing.
As per the usual guidelines, but add --download-f-blas-lapack=1
Petsc4py as per usual guidelines.
Problem: Need libbz2 and rebuild Python with bz2 module. Argh.
Get from http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz and build.
pip install bzr+http://bazaar.launchpad.net/~florian-rathgeber/ufc/python-setup#egg=ufc_utils
pip install https://sourcesup.renater.fr/frs/download.php/2309/ScientificPython-2.8.tar.gz
pip install bzr+http://bazaar.launchpad.net/~fiat-core/fiat/main#egg=fiat
Codepy does not support the Intel compiler. Use the OP2/codepy fork.
#!/bin/bash
# Base directory for work
export BASE=/home/grm08/ics-paper
# CUDA/OpenCL
module load opencl/amd-2.7
module load cuda
# CMake
module load cmake/2.8.9
# ICC and MPI
module load intel-suite/11.1
module load mpi
# Don't accidentally use GNU compilers
function nognu() { echo "NO GNU" 1>&2; false; }
alias gcc=nognu
alias g++=nognu
alias gfortran=nognu
# Executables on installed path
export INSTALL_PATH=$BASE/install
export PATH=$INSTALL_PATH/bin:$PATH
export LD_LIBRARY_PATH=$INSTALL_PATH/lib:$LD_LIBRARY_PATH
# Extra include and lib dirs
export INCLUDE_PATH=$INSTALL_PATH/include:$INCLUDE_PATH
export LIBRARY_PATH=$INSTALL_PATH/lib:$LIBRARY_PATH
export CC=mpicc
export CXX=mpicxx
export F90=mpif90
export F77=mpif77
export FC=mpifc
# OpenCL include and lib dirs
export INCLUDE_PATH=/apps/opencl/2.0/usr/include:$INCLUDE_PATH
export LIBRARY_PATH=/apps/opencl/2.0/usr/lib64:$LIBRARY_PATH
export LD_LIBRARY_PATH=/apps/opencl/2.0/usr/lib64:$LD_LIBRARY_PATH
# OP2
export OP2_DIR=$BASE/OP2-Common/op2
export LD_LIBRARY_PATH=$OP2_DIR/c/lib:$LD_LIBRARY_PATH
# PETSc
export PETSC_CONFIGURE_OPTIONS="--with-fortran-interfaces=1 --with-c++-support --with-openmp --download-f-blas-lapack=1"
export PETSC_DIR=/home/grm08/ics-paper/install/lib/python2.7/site-packages/petsc
# PyOP2
export PYTHONPATH=$BASE/PyOP2:$PYTHONPATH
# FEniCS
export PYTHONPATH=$BASE/ffc/pyop2:$PYTHONPATH
export PYTHONPATH=$BASE/ufl/trunk:$PYTHONPATH
# Stupid hack
export LD_PRELOAD=/apps/intel/cc/11.1.073/lib/intel64/libiomp5.so
#PBS -N buildfluidity
#PBS -l walltime=1:00:00
#PBS -l select=1:ncpus=1:mem=1800mb
source /home/grm08/ics-v2/e.sh
export LIBS="-L${ZOLTAN_HOME}/lib -L${VTK_LIBS} -L${NETCDF_HOME}/lib -L${UDUNITS_HOME}/lib -L${PARMETIS_HOME}/lib -L/usr/X11R6/lib64/ -L${MKL_HOME}/lib/64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core"
export CPPFLAGS="-I${ZOLTAN_HOME}/include -I${VTK_INCLUDE} -I${NETCDF_HOME}/include -I${UDUNITS_HOME}/include -I${PARMETIS_HOME}/include"
export ZOLTAN_DEPS="-L${ZOLTAN_HOME}/lib -L${LIBSCOTCH_HOME}/lib -lpthread -lptscotch -lpatoh -lptscotcherr -lptscotcherrexit -lscotchmetis -lscotch"
export FLUIDITY_DIR=${BASE}/fluidity
export PYTHONPATH=${FLUIDITY_DIR}/python:$PYTHONPATH
cd ${FLUIDITY_DIR}
./configure --disable-petsc-fortran-modules --enable-2d-adaptivity > compile.log 2>&1 && make clean >> compile.log 2>&1 && make >> compile.log 2>&1 && make fltools >> compile.log 2>&1
BOOST_INC_DIR = ['/apps/boost/1.52.0/build/boost_1_52_0']
BOOST_LIB_DIR = ['/apps/boost/1.52.0/build/boost_1_52_0/stage/lib']
BOOST_COMPILER = 'gcc43'
USE_SHIPPED_BOOST = False
BOOST_PYTHON_LIBNAME = ['boost_python']
BOOST_THREAD_LIBNAME = ['boost_thread']
CUDA_TRACE = False
CUDA_ROOT = '/apps/cuda/5.0.35'
CUDA_ENABLE_GL = False
CUDA_ENABLE_CURAND = True
CUDADRV_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CUDADRV_LIBNAME = ['cuda']
CUDART_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CUDART_LIBNAME = ['cudart']
CURAND_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CURAND_LIBNAME = ['curand']
CXXFLAGS = []
LDFLAGS = []