<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body text="#FFFFFF" bgcolor="#000000">
<p><font face="Helvetica, Arial, sans-serif">Dear community.<br>
</font></p>
<p><font face="Helvetica, Arial, sans-serif">I'm a newbie in
Quantum-ESPRESSO, and I got stuck in a situation a little
confusing for me. Long story short: I was trying to compile
QE-6.4.1 recently, and as far as I can see everything is ok, but
when I run it I see that the subspace diagonalization is placed
as serial and not parallel.</font></p>
<p><font face="Helvetica, Arial, sans-serif">So, that being said
here is some details of what I have done so far:</font></p>
<p><font face="Helvetica, Arial, sans-serif">After I downloaded the
source and applied the patch, I loaded the necessary modules (my
case were: intel2019, mkl2019, intelmpi2019, openmpi version
2.1.5 is already the default loaded openmpi in my working
environment), and I set the environment variables below wihthin
the qe-641 directory: <br>
<br>
ESPRESSO_ROOT=$(pwd)<br>
export SCALAPACK_LIBS="-L${MKLROOT}/lib/intel64
-libmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64
-lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -lpthread -lm"<br>
export LAPACK_LIBS="-L${MKLROOT}/lib/intel64
-lmkl_scalapack_lp64 -libmkl_intel_lp64
-lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_core
-lmkl_intel_thread -lpthread -lm"<br>
export BLAS_LIBS="-L${MKLROOT}/lib/intel64 -lmkl_scalapack_lp64
-lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_core
-lmkl_intel_thread -lpthread -lm"<br>
export CC="icc -D_Float128=__float128"<br>
export FC=ifort <br>
export F77=ifort<br>
export MPIF90=mpiifort<br>
<br>
Then fired configure as:<br>
<br>
./configure --prefix=${ESPRESSO_ROOT} --enable-parallel
--enable-openmp --with-scalapack=intel<br>
<br>
Finally I got a make.inc that seems like this (only changed
CFLAGS and FFLAGS by adding -xCORE-AVX512):<br>
<br>
IFLAGS = -I$(TOPDIR)/include -I$(TOPDIR)/FoX/finclude
-I$(TOPDIR)/S3DE/iotk/include/ -I${MKLROOT}/include<br>
[...]<br>
DFLAGS = -D__DFTI -D__MPI -D__SCALAPACK<br>
[...]<br>
MPIF90 = mpiifort<br>
F90 = ifort<br>
CC = icc -D_Float128=__float128<br>
F77 = ifort<br>
[...]<br>
CFLAGS = -O3 -xCORE-AVX512 $(DFLAGS) $(IFLAGS)<br>
F90FLAGS = $(FFLAGS) -nomodule -qopenmp -fpp $(FDFLAGS)
$(CUDA_F90FLAGS) $(IFLAGS) $(MODFLAGS) <br>
FFLAGS = -O2 -xCORE-AVX512 -assume byterecl -g
-traceback -qopenmp -I${MKLROOT}/include/fftw<br>
<br>
So, after that I fired 'make all' and 'make install' and
everything went fine without any error message.<br>
Hence I prepared the following script to run a calculation:<br>
<br>
export I_MPI_HYDRA_BOOTSTRAP_EXEC=/opt/pbs/bin/pbs_tmrsh<br>
export I_MPI_HYDRA_BOOTSTRAP=rsh<br>
export FI_PROVIDER=tcp<br>
export OMP_NUM_THREADS=1<br>
export MKL_NUM_THREADS=1<br>
export KMP_AFFINITY=scatter,granularity=fine,1<br>
nk=V<br>
ni=W<br>
nb=X<br>
nd=Y<br>
nt=Z<br>
N=`echo "$ni*$nk*$nb" | bc -l`<br>
<br>
CMD_QE=/opt/sw/qe-641/bin/pw<br>
<br>
mpiexec.hydra -n $N $CMD_QE/pw.x -npool $nk -ndiag $nd -ntg $nt
-ni $ni -nb $nb < $CALC_DIR/input.in >&
$CALC_DIR/output.out<br>
<br>
Everything that I run using this script goes fine, but the
question is still: why subspace diagonalization is serial?<br>
I have played a lot by changing -nk -nb -nd -nt values without
success.<br>
I already have installed the QE-6.1.0 version and I took almost
the same approach as here and everything works fine, this is, I
get the subspace diagonalization to work in parallel.<br>
</font></p>
<p><font face="Helvetica, Arial, sans-serif">The reason I would like
to try QE-6.4.1 is due the grimme-d3 method it has implemented
(and because I would like to try GPU in near future for learning
purposes).</font></p>
<p><font face="Helvetica, Arial, sans-serif">I apologize for my
question, and I will be glad to get some help from you to better
understand what is missing and where I'm doing it wrong.</font></p>
<p><font face="Helvetica, Arial, sans-serif">Nevertheless, if I
missed any information and there is something that I should
tell, please, just name it and I will be happy to clarify
everything as much as I can.</font></p>
<p><font face="Helvetica, Arial, sans-serif">Thanks,<br>
</font></p>
<div class="moz-signature">-- <br>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title></title>
<img moz-do-not-send="false"
src="cid:part1.CEDADE9F.E1DE8EAB@gmail.com" alt="UNICAMP
Materials Simulation Lab" width="406" height="116"><br>
___________________________________________________<br>
PhD. Student Manoel Victor Frutuoso Barrionuevo<br>
Institute of Chemistry - University of Campinas<br>
Campinas - 13083-970 - São Paulo - Brasil<br>
<a moz-do-not-send="true" href="umsl.iqm.unicamp.br">umsl.iqm.unicamp.br</a><br>
+55(17)-99723-3966<br>
</div>
</body>
</html>