crusher

Crusher at Oak Ridge National Laboratory

[TOC]

Download

SLATE git repo can be cloned as follows:

git clone --recursive https://bitbucket.org/icl/slate-dev.git

Installation

SLATE can be installed on Crusher using the following steps. The current directory is assumed to be SLATE source code.

module load cce/14.0.3
module load cray-mpich/8.1.19
module load rocm/5.2.0
module load craype-accel-amd-gfx90a
export MPICH_GPU_SUPPORT_ENABLED=1
export CPATH=${ROCM_PATH}/include
export LIBRARY_PATH=${ROCM_PATH}/lib:$LIBRARY_PATH
export LD_LIBRARY_PATH="${CRAY_LD_LIBRARY_PATH}:${LD_LIBRARY_PATH}"
top=$PWD
cat > make.inc << END
CXX=CC
FC=ftn
CXXFLAGS=-I${ROCM_PATH}/include -craype-verbose
LDFLAGS=-L${ROCM_PATH}/lib -craype-verbose
blas=libsci
gpu_backend=hip
hip_arch=gfx90a
mpi=1
END
make -j 20 install prefix=${top}/install

Running

The following steps compile and run slate_lu.cc example:

cd examples
CC -Wl,-rpath,${top}/../install/lib/ -L../install/lib/ -I../install/include/ -L/opt/rocm-5.2.0/lib -L/opt/rocm-5.2.0/include -fopenmp -D__HIP_PLATFORM_HCC__ -DSLATE_NO_CUDA -lblaspp -llapackpp -lslate -o slate_lu slate_lu.cc

These commands can be used to allocate an interactive session and run SLATE's LU factorization example:

salloc -A <project id>_crusher -t 0:10:0 --nodes 1
srun -n 4 ./slate_lu

This command can be used to run SLATE's LU factorization example:

srun -A <project id>_crusher -t 1 -N 1 -n 4 ./slate_lu

Expected output:

gesv n 5000 nb 256 pxq 2x2 residual 1.05e-20 tol 2.22e-16 sec 3.76e+00 PASS