<div dir="ltr"><div><div><div>Paolo,<br><br></div> Thank you for the response. I am running the code using <br><br>module purge<br>export OMP_NUM_THREADS=1<br>export PSM_RANKS_PER_CONTEXT=4<br>module add compilers/intel/2015_cluster_xe<br><br>#mpirun -ppn ${SLURM_NTASKS_PER_NODE} -n ${SLURM_NTASKS} /home/r/rch/espresso/openMP_EXE/pw.x -npools 4 -ntg 2 -in <a href="http://file.in">file.in</a><br><br></div>with <br><br>#SBATCH --ntasks-per-node=24<br>#SBATCH -N 2<br><br></div>If I remove the -npools 4 -ntg flags and rerun, I receive the same error though it is not printed 48 times as I would expect but only twice. If I take your suggestion and change the mpirun line to <br><div><div><br>mpirun -ppn ${SLURM_NTASKS_PER_NODE} -n ${SLURM_NTASKS} /home/r/rch/espresso/openMP_EXE/pw.x -nd 1 -in <a href="http://file.in">file.in</a><br><br></div><div>it runs fine with 48 processors. Combining all these flags with <br><br>mpirun -ppn ${SLURM_NTASKS_PER_NODE} -n ${SLURM_NTASKS} /home/r/rch/espresso/openMP_EXE/pw.x -nd 1 -npools 4 -ntg 2 -in <a href="http://file.in">file.in</a><br><br></div><div>also works well. Including the -nd 1 flag seems to have fixed the problem and it also runs well with 96 processors and using different k-point and task group parallelizations. Thank you for telling me about that, I have never heard of that flag. If you would like for me to provide any additional information about the way I have ran the calculations for the PW_forum record, please let me know.<br><br></div><div> Thank you, Ryan Herchig<br><br></div><div> University of South Florida, Tampa FL, Department of Physics<br></div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Dec 6, 2016 at 11:19 AM, Paolo Giannozzi <span dir="ltr"><<a href="mailto:p.giannozzi@gmail.com" target="_blank">p.giannozzi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>I am not convinced that the problem you mention is the same as yours. In order to figure out if the problem arises from Scalapack, you should remove __SCALAPACK from DFLAGS and recompile: the code will use (much slower) internal routines for parallel dense-matrix diagonalization. You may also try to run with no dense-matrix diagonalization (-nd 1, not sure it is honored though). <br><br>You should also report how your are running your code and, if using exotic parallelizations like "band groups" (-nb N), check if the problem you have is related to its usage<br><br></div>Paolo<br><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Thu, Dec 1, 2016 at 11:37 PM, Ryan Herchig <span dir="ltr"><<a href="mailto:rch@mail.usf.edu" target="_blank">rch@mail.usf.edu</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><div><div><div><div>Hello all,<br><br></div> I am running pw.x in Quantum Espresso version 5.4.0, however if I try and run the job using more than 2 nodes with 8 cores each I receive the following error :<br><br>Fatal error in PMPI_Group_incl: Invalid rank, error stack:<br>PMPI_Group_incl(185)..........<wbr>...: MPI_Group_incl(group=0x8800000<wbr>4, n=4, ranks=0x2852700, new_group=0x7fff57564668) failed<br>MPIR_Group_check_valid_ranks(2<wbr>53): Invalid rank in rank array at index 3; value is 33 but must be in the range 0 to 31<br><br></div><div>I am building/running on a local cluster maintained by the University I attend. The specifications for the nodes are 2 x Intel Xeon E5-2670 (Eight Core)
32GB
QDR InfiniBand.
I found in a previous thread<br><br><a href="https://www.mail-archive.com/pw_forum@pwscf.org/msg27702.html" target="_blank">https://www.mail-archive.com/p<wbr>w_forum@pwscf.org/msg27702.htm<wbr>l</a><br><br></div>involving espresso-5.3.0 where another user seemed to be experiencing the same issue where it was determined that "The problem is related to the obscure hacks needed to convince Scalapack to work in a subgroup of processors." The suggestion in this post was to change a line in Modules/mp_global.f90 and recompile. However I am running spin-collinear vdW-DF calculations which requires at least version 5.4.0 I believe and the lines in the subroutine found in mp_global.f90 has changed; furthermore following the suggestion of the previous post does not fix the issue. It instead produces the following compilation error :<br><br>mp_global.f90(97): error #6631: A non-optional actual argument must be present when invoking a procedure with an explicit interface. [NPARENT_COMM]<br> CALL mp_start_diag ( ndiag_, intra_BGRP_comm )<br>---------^<br>mp_global.f90(97): error #6631: A non-optional actual argument must be present when invoking a procedure with an explicit interface. [MY_PARENT_ID]<br> CALL mp_start_diag ( ndiag_, intra_BGRP_comm )<br>---------^<br>compilation aborted for mp_global.f90 (code 1)<br><br><br>Does this problem with the ScaLAPACK libraries persist in the newer versions or could these errors have a separate origin? Possibly something I am doing wrong during the build? I have included the make.sys that I am using for "make pw" below. If the error is due to the ScaLAPACK libraries, is there a workaround which could allow the use of additional processors when running calculations? Thank you in advance.<br><br></div> Thank you, Ryan Herchig<br><br></div> University of South Florida, Department of Physics<br><div><div><br><br>.SUFFIXES :<br>.SUFFIXES : .o .c .f .f90<br><br>.f90.o:<br> $(MPIF90) $(F90FLAGS) -c $<<br><br># .f.o and .c.o: do not modify<br><br>.f.o:<br> $(F77) $(FFLAGS) -c $<<br><br>.c.o:<br> $(CC) $(CFLAGS) -c $<<br><br>TOPDIR = /work/r/rch/espresso-5.4.0<br><br>MANUAL_DFLAGS =<br>DFLAGS = -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK<br>FDFLAGS = $(DFLAGS) $(MANUAL_DFLAGS)<br><br>IFLAGS = -I../include -I/apps/intel/2015/composer_xe<wbr>_2015.3.187/mkl/include:/apps/<wbr>intel/2015/composer_xe_2015.3.<wbr>187/tbb/include<br><br>MOD_FLAG = -I<br><br>MPIF90 = mpif90<br>#F90 = ifort<br>CC = icc<br>F77 = ifort<br><br>CPP = cpp<br>CPPFLAGS = -P -C -traditional $(DFLAGS) $(IFLAGS)<br><br>CFLAGS = -O3 $(DFLAGS) $(IFLAGS)<br>F90FLAGS = $(FFLAGS) -nomodule -fpp $(FDFLAGS) $(IFLAGS) $(MODFLAGS)<br>FFLAGS = -O2 -assume byterecl -g -traceback<br><br>FFLAGS_NOOPT = -O0 -assume byterecl -g -traceback<br><br>FFLAGS_NOMAIN = -nofor_main<br><br>LD = mpif90<br>LDFLAGS = <br>LD_LIBS = <br><br>BLAS_LIBS = -lmkl_intel_lp64 -lmkl_sequential -lmkl_core<br>BLAS_LIBS_SWITCH = external<br><br>LAPACK_LIBS = -L/apps/intel/2015/composer_xe<wbr>_2015.3.187/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core<br>LAPACK_LIBS_SWITCH = external<br><br>ELPA_LIBS_SWITCH = disabled<br>SCALAPACK_LIBS = -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_ilp64<br><br>FFT_LIBS = -L/apps/intel/2015/composer_xe<wbr>_2015.3.187/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_sequential -lmkl_core<br><br>MPI_LIBS = <br><br>MASS_LIBS = <br><br>AR = ar<br>ARFLAGS = ruv<br><br>RANLIB = ranlib<br><br>FLIB_TARGETS = all<br><br>LIBOBJS = ../clib/clib.a ../iotk/src/libiotk.a<br>LIBS = $(SCALAPACK_LIBS) $(LAPACK_LIBS) $(FFT_LIBS) $(BLAS_LIBS) $(MPI_LIBS) $(MASS_LIBS) $(LD_LIBS)<br><br>WGET = wget -O<br><br>PREFIX = /work/r/rch/espresso-5.4.0/EXE<br><br><br><br><br><br></div></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/listi<wbr>nfo/pw_forum</a><span class="HOEnZb"><font color="#888888"><br></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_2816238446729906723gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>Phone <a href="tel:+39%200432%20558216" value="+390432558216" target="_blank">+39-0432-558216</a>, fax <a href="tel:+39%200432%20558222" value="+390432558222" target="_blank">+39-0432-558222</a><br><br></div></div></div></div></div>
</font></span></div>
<br>______________________________<wbr>_________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/<wbr>listinfo/pw_forum</a><br></blockquote></div><br></div>