Skip to content
This repository has been archived by the owner on Nov 27, 2024. It is now read-only.

Graham's notes on CX1

Jump to bottom
gmarkall edited this page Jan 29, 2013 · 35 revisions

This page is created for notes/potential improvements to the guidelines without disturbing Fabio's original notes.

These are not guidelines to follow specifically - please don't remove info from this page either.

ICC

Need to compile things with icc.

Python and pip

Variant 1: Compiling python

Download and compile python 2.7.3, install to home dir.

Needs bzip2 available:

wget http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz
tar xpzf bzip2-1.0.6.tar.gz
cd bzip2-1.0.6
make
make -f Makefile_libbz2_so

Make sure the .so and .h files are found by Python configure.

In order to install pip on CX1, you can use:

curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py
curl -O https://raw.github.com/pypa/pip/master/contrib/get-pip.py
python get-pip.py
cd build/pip
python setup.py build
python setup.py install

Variant 2: use a virtualenv

Install virtualenv (if you don't have it already):

git clone git://github.com/pypa/virtualenv.git
cd virtualenv/
python setup.py install --user

Create a virtualenv in your PyOP2 tree:

python -m virtualenv --distribute .venv

and activate it:

. .venv/bin/activate

You can now use pip to install packages in your virtualenv!

To leave the virutalenv you can use deactivate.

Using pip

Installing dependencies:

pip install cython decorator pyyaml pytest codepy ply jinja2 mako
pip install https://launchpad.net/instant/1.0.x/1.0.0/+download/instant-1.0.0.tar.gz
pip install hg+https://bitbucket.org/fr710/pycparser#egg=pycparser-2.08

Numpy/Scipy

Used guidelines at: http://software.intel.com/en-us/articles/numpy-scipy-with-mkl/

Used NP 1.6.2 and Scipy 0.9.0. Needed this fix: http://mail.scipy.org/pipermail/scipy-user/2011-July/029908.html

PyOpenCL

you need to

module load opencl

Clone the PyOpenCL repo and edit setup.py lines:

        IncludeDir("CL", []),
        LibraryDir("CL", []),

to

        IncludeDir("CL", ['/apps/opencl/2.0/usr/include']),
        LibraryDir("CL", ['/apps/opencl/2.0/usr/lib64']),

Then install with

python setup.py build
python setup.py install

OP2-Common

Installation as per the README

PyCUDA

git clone http://git.tiker.net/trees/pycuda.git
cd pycuda
git submodule init
git submodule update
python configure.py --no-use-shipped-boost
python setup.py build
python setup.py install

Codepy

Use the codepy from the OP2 repository. Do not use the CX1 branch, this was just for experimentation. Use the master branch.

In order to get it to see CUSP, I added this at line 457 of pyop2/cuda.py:

    nvcc_toolchain.add_library('cusp',['<path to cusp>'],[],[])

PETSC

As per the usual guidelines, but add --download-f-blas-lapack=1 Petsc4py as per usual guidelines.

FEniCS bits and bobs

Bzip2

Problem: Need libbz2 and rebuild Python with bz2 module. Argh.

Get from http://bzip.org/1.0.6/bzip2-1.0.6.tar.gz and build.

FEniCS packages

pip install bzr+http://bazaar.launchpad.net/~florian-rathgeber/ufc/python-setup#egg=ufc_utils
pip install https://sourcesup.renater.fr/frs/download.php/2309/ScientificPython-2.8.tar.gz
pip install bzr+http://bazaar.launchpad.net/~fiat-core/fiat/main#egg=fiat

Codepy

Codepy does not support the Intel compiler. Use the OP2/codepy fork.

Graham's current environment

#!/bin/bash

# Modules
module load intel-suite/11.1
module load mpi
module load cmake/2.8.9
module load swig/2.0.2
module load gmsh/2.5.0
module load bzr/2.5.0
module load cuda/5.0.35
module load boost/1.52.0

# Fluidity modules
module load vtk/5.8.0
module load netcdf
module load udunits
module load parmetis
module load zoltan
module load libscotch

# Base dir
export BASE=/home/grm08/ics-v2

# In lieu of module load Python/2.7.3
export PYTHON_HOME=$BASE/env
export PYTHON_VERSION=2.7.3
export PATH=$PYTHON_HOME/bin:$PATH
export LD_LIBRARY_PATH=/apps/python/2.7.3/lib:$LD_LIBRARY_PATH
export PYTHONPATH=$PYTHON_HOME/lib/python2.7/site-packages:$PYTHONPATH
export LD_PRELOAD=/usr/lib64/libz.so:/usr/lib64/libxml2.so:$LD_PRELOAD

# Use virtualenv
source $BASE/env/bin/activate

# PETSc
export PETSC_DIR=$BASE/env/lib/python2.7/site-packages/petsc

# OP2
export OP2_DIR=$BASE/OP2-Common/op2

# PyOP2
export PYTHONPATH=$BASE/PyOP2:$PYTHONPATH

# UFL
export PYTHONPATH=$BASE/ufl/trunk:$PYTHONPATH

# FFC
export PYTHONPATH=$BASE/ffc/pyop2:$PYTHONPATH

# Instant
export PYTHONPATH=$BASE/instant/trunk:$PYTHONPATH

# FIAT
export PYTHONPATH=$BASE/fiat/trunk:$PYTHONPATH

# UFC
export PYTHONPATH=$BASE/ufc/trunk/src/utils/python:$PYTHONPATH

# BLAS/LAPACK for Scipy
export BLAS=/apps/intel/cc/11.1.073/mkl/lib/intel64/libblas.so
export LAPACK=/apps/intel/cc/11.1.073/mkl/lib/intel64/liblapack.so

# Fluidity
export PYTHONPATH=$BASE/fluidity/python:$PYTHONPATH
export LD_LIBRARY_PATH=/home/grm08/ics-v2/fluidity/lib:$LD_LIBRARY_PATH

# PyCUDA
export PYTHONPATH=$BASE/pycuda/build/lib.linux-x86_64-2.7:$PYTHONPATH

# Seems to be required for numpy to work
export LD_PRELOAD=/apps/intel/cc/11.1.073/lib/intel64/libiomp5.so:$LD_PRELOAD

PBS script to build fluidity

#PBS -N buildfluidity
#PBS -l walltime=1:00:00
#PBS -l select=1:ncpus=1:mem=1800mb

source /home/grm08/ics-v2/e.sh

export LIBS="-L${ZOLTAN_HOME}/lib -L${VTK_LIBS} -L${NETCDF_HOME}/lib -L${UDUNITS_HOME}/lib -L${PARMETIS_HOME}/lib -L/usr/X11R6/lib64/ -L${MKL_HOME}/lib/64/ -lmkl_intel_lp64 -lmkl_sequential -lmkl_core"
export CPPFLAGS="-I${ZOLTAN_HOME}/include -I${VTK_INCLUDE} -I${NETCDF_HOME}/include -I${UDUNITS_HOME}/include -I${PARMETIS_HOME}/include"
export ZOLTAN_DEPS="-L${ZOLTAN_HOME}/lib -L${LIBSCOTCH_HOME}/lib -lpthread -lptscotch -lpatoh -lptscotcherr -lptscotcherrexit -lscotchmetis -lscotch"

export FLUIDITY_DIR=${BASE}/fluidity
export PYTHONPATH=${FLUIDITY_DIR}/python:$PYTHONPATH

cd ${FLUIDITY_DIR}

./configure --disable-petsc-fortran-modules --enable-2d-adaptivity > compile.log 2>&1 && make clean >> compile.log 2>&1 && make >> compile.log 2>&1 && make fltools >> compile.log 2>&1

~/.aksetup-defaults.py


BOOST_INC_DIR = ['/apps/boost/1.52.0/build/boost_1_52_0']
BOOST_LIB_DIR = ['/apps/boost/1.52.0/build/boost_1_52_0/stage/lib']
BOOST_COMPILER = 'gcc43'
USE_SHIPPED_BOOST = False
BOOST_PYTHON_LIBNAME = ['boost_python']
BOOST_THREAD_LIBNAME = ['boost_thread']
CUDA_TRACE = False
CUDA_ROOT = '/apps/cuda/5.0.35'
CUDA_ENABLE_GL = False
CUDA_ENABLE_CURAND = True
CUDADRV_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CUDADRV_LIBNAME = ['cuda']
CUDART_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CUDART_LIBNAME = ['cudart']
CURAND_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib64']
CURAND_LIBNAME = ['curand']
CXXFLAGS = []
LDFLAGS = []
Clone this wiki locally