[Pw_forum] Hybrid MPI OpenMP

[Ben Palmer]

Hi Everyone,

I have compiled QE 5.0.2 on a computer with AMD interlagos processors, 
using the acml, compiling with openmp enabled, and submitting jobs with 
PBS.  I've had a speed up using 2 openmp threads per mpi process.

I've been trying to do the same on another computer, that has MOAB as 
the scheduler, E5 series xeon processors (E5-2660) and uses the Intel 
MKL (E5-2660).  I'm pretty sure hyperthreading has been turned off, as 
each node has two sockets and 16 cores in total.

I've seen a slow down in performance using OpenMP and MPI, but have read 
that this might be the case in the documentation.  I'm waiting in the 
computer's queue to run the following:

#!/bin/bash
#MOAB -l "nodes=2:ppn=16"
#MOAB -l "walltime=0:01:00"
#MOAB -j oe
#MOAB -N pwscf_calc
#MOAB -A readmsd02
#MOAB -q bbtest
cd "$PBS_O_WORKDIR"
module load apps/openmpi/v1.6.3/intel-tm-ib/v2013.0.079
export PATH=$HOME/bin:$PATH
export OMP_NUM_THREADS=2
mpiexec -np 16 -x OMP_NUM_THREADS=2 -npernode 8 -bind-to-socket 
-display-map -report-bindings pw_openmp_5.0.2.x -in benchmark2.in > 
benchmark2c.out

I just wondered if anyone had any tips on the settings or flags for 
hybrid MPI/OpenMP with the E5 Xeon processors?

All the best,

Ben Palmer
Student @ University of Birmingham, UK