<!DOCTYPE html><html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8" /></head><body>Dear All,<div><br></div><div>I am trying to do some calculations on simple silicon with GPU support as a test case before proceeding serious work. </div><div><br></div><div>QE-GPU has been installed by intel cluster suite 13, cuda 5.5 and latest svn commits of QE and QE-GPU. </div><div><br></div><div><p style="font-size: 15px; font-family: Courier;">MANUAL_DFLAGS = -D__MPICH_SKIP_MPICXX -D__ISO_C_BINDING -D__VERBOSE<br></p><p style="font-size: 15px; font-family: Courier;">DFLAGS = -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -D__ELPA -D__CUDA -D__PHIGEMM -D__OPENMP $(MANUAL_DFLAGS)<br></p><p style="font-size: 15px; font-family: Courier;"><br><br></p><p style="font-size: 15px; font-family: Courier;">I have for 4 nodes. Each one has 24 cores and 2 GPU support so totally 96 cores + 8 GPU. <br></p><p style="font-size: 15px; font-family: Courier;"><br><br></p><p style="font-size: 15px; font-family: Courier;">I haven't got any problem for 1 node. I did it. When I passed to bigger than 1 node I am getting warnings, for example,<br></p><p style="font-size: 15px; font-family: Courier;">***WARNING: unbalanced configuration (1 MPI per node, 2 GPUs per node)<br></p><p style="font-size: 15px; font-family: Courier;">
<br></p><p style="font-size: 15px; font-family: Courier;">***WARNING: unbalanced configuration (13 MPI per node, 2 GPUs per node)<br></p><p style="font-size: 15px; font-family: Courier;"><br><br></p><p style="font-size: 15px; font-family: Courier;">So, I am in trouble for sharing the cores between MPI and threads for OpenMP. <br></p><p style="font-size: 15px; font-family: Courier;"><br><br></p><p style="font-size: 15px; font-family: Courier;">Question: How should I share the cores between MPI and OpenMP threads? <br></p><p style="font-size: 15px; font-family: Courier;"><br><br></p><p style="font-size: 15px; font-family: Courier;">Here is my case:<br></p><p style="font-size: 15px; font-family: Courier;"><br><br></p><p style="font-size: 15px; font-family: Courier;">#SBATCH -N 4<br></p><p style="font-size: 15px; font-family: Courier;">#SBATCH -n 4<br></p><p style="font-size: 15px; font-family: Courier;">#SBATCH --gres=gpu:2<br></p><p style="font-size: 15px; font-family: Courier; min-height: 18px;"><br><br></p><p style="font-size: 15px; font-family: Courier;">module load centos6.4/lib/cuda/5.5<br></p><p style="font-size: 15px; font-family: Courier; min-height: 18px;"><br><br></p><p style="font-size: 15px; font-family: Courier;"> export OMP_NUM_THREADS=24<br></p><p style="font-size: 15px; font-family: Courier;"> export MKL_NUM_THREADS=$OMP_NUM_THREADS<br></p><p style="font-size: 15px; font-family: Courier;"> export PHI_DGEMM_SPLIT=0.975<br></p><p style="font-size: 15px; font-family: Courier;"> export PHI_ZGEMM_SPLIT=0.975<br></p><p style="font-size: 15px; font-family: Courier; min-height: 18px;"><br><br></p><p style="font-size: 15px; font-family: Courier;">
<br></p><p style="font-size: 15px; font-family: Courier;">mpirun pw-gpu.x -nd 1 < si.in > out.GPU<br></p><p style="font-size: 15px; font-family: Courier;"><br><br></p><p style="font-size: 15px; font-family: Courier;">Sincerely Yours,<br></p><p style="font-size: 15px; font-family: Courier;"><br><br></p><p style="font-size: 15px; font-family: Courier;"> Mutlu.<br><br></p><div><div class="crea-font-wrapper" style="font-family: Tahoma; font-size: 16px;">------------------------------------------</div><div class="crea-font-wrapper" style="font-family: Tahoma; font-size: 16px;">Mutlu ÇOLAKOĞULLARI</div><div class="crea-font-wrapper" style="font-family: Tahoma; font-size: 16px;">Trakya Universitesi </div><div class="crea-font-wrapper" style="font-family: Tahoma; font-size: 16px;">Fen Fakultesi</div><div class="crea-font-wrapper" style="font-family: Tahoma; font-size: 16px;">Fizik Bolumu</div><div class="crea-font-wrapper" style="font-family: Tahoma; font-size: 16px;">22030 Merkez-EDIRNE/TURKEY</div></div></div></body></html>