[Pw_forum] QE-5.4.0 on hpc cluster w gfortran + MKL and other

[Ari P Seitsonen]

Dear Eugenia,

   If you have not your problem yet, here a recent reply from Paolo G.:

http://qe-forge.org/pipermail/pw_forum/2016-October/111293.html

Basically you have to include 
'/opt/intel/composerxe-2013.0.080/mkl/include' (or similar) in the 
directories where the module files are looked for, 'IFLAGS'. Hopefully 
this helps.

     Greetings from Paris,

        apsi

-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-=*=-
   Ari Paavo Seitsonen / Ari.P.Seitsonen at iki.fi / http://www.iki.fi/~apsi/
     Ecole Normale Supérieure (ENS), Département de Chimie, Paris
     Mobile (F) : +33 789 37 24 25    (CH) : +41 79 71 90 935


On Tue, 1 Nov 2016, eugenia wetrowa wrote:

> Dear all, please help. I'm trying to build parallel QE-5.4.0 on hpc cluster.  System parameters:
> 20 nodes with
> 2* E5-2680 Intel Xeon CPU (8 n), 128GiB RAM 300 GiB HDD
> arch: x86_64
> Gentoo/Linux 3.2.59
> GCC compilers, gfortran 4.8
> OpenMPI, mpich-2 mpi-mpich2-1_5-x86_64
> 
> Autoconfig detects compilers and parallel environment correctly but  links libraries:
> 
> BLAS_LIBS      = -L/opt/intel/composerxe-2013.0.080/mkl/lib/intel64 -lmkl_gf_lp64  -lmkl_sequential -lmkl_core
> BLAS_LIBS_SWITCH = external
> LAPACK_LIBS    =
> LAPACK_LIBS_SWITCH = external
> ELPA_LIBS_SWITCH = disabled
> SCALAPACK_LIBS = -lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
> FFT_LIBS       = -lfftw3
> MPI_LIBS       =
> MASS_LIBS      =
> AR             = ar
> ARFLAGS        = ruv
> RANLIB         = ranlib
> FLIB_TARGETS   = all
> LIBOBJS        = ../clib/clib.a ../iotk/src/libiotk.a
> LIBS           = $(SCALAPACK_LIBS) $(LAPACK_LIBS) $(FFT_LIBS) $(BLAS_LIBS) $(MPI_LIBS) $(MASS_LIBS) $(LD_LIBS)
> 
> In this configuration "make" fails:
> 
> _ _ _ _ _ _
> test -d bin || mkdir bin
> ( cd FFTXlib ; make TLDEPS= all || exit 1 )
> make[1]: Entering directory `/home/osvasilyev/QE/FFTXlib'
> mpif90 -O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__DFTI -D__MPI -D__PARA -D__SCALAPACK  -I../include -I/include  -c fft_types.f90
> Warning: Nonexistent include directory "/include"
> mpif90 -O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__DFTI -D__MPI -D__PARA -D__SCALAPACK  -I../include -I/include  -c scatter_mod.f90
> Warning: Nonexistent include directory "/include"
> mpif90 -O3 -g -x f95-cpp-input -D__GFORTRAN -D__STD_F95 -D__DFTI -D__MPI -D__PARA -D__SCALAPACK  -I../include -I/include  -c fft_scalar.f90
> Warning: Nonexistent include directory "/include"
> In file included from fft_scalar.f90:27:0:
> fft_scalar.DFTI.f90:20:0: fatal error: mkl_dfti.f90: No such file or directory
>  #include "mkl_dfti.f90"
>  ^
> compilation terminated.
> make[1]: *** [fft_scalar.o] Error 1
> make[1]: Leaving directory `/home/osvasilyev/QE/FFTXlib'
> make: *** [libfft] Error 1
> _ _ _ _ _ _ _
> 
> Then i tryed to build QE with MKL libraries following the Intel's manual, linking explicitlly ./configure --LIBDIRS="
> -L/opt/intel/composerxe-2013.0.080/mkl/lib/intel64 -Wl, -lmkl_scalapack_ilp64 -lmkl_gf_ilp64 -lmkl_gnu_thread -lmkl_core
> -lmkl_blacs_intelmpi_ilp64 -lgomp -lpthread -lm -ldl" .
> Configure says
> "The following libraries have been found:
>   BLAS_LIBS=-L/opt/intel/composerxe-2013.0.080/mkl/lib/intel64 -lmkl_gf_lp64  -lmkl_sequential -lmkl_core
>   LAPACK_LIBS=
>   SCALAPACK_LIBS=-lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64
>   FFT_LIBS=-lfftw3
> parallel environments detected successfully"
> 
> But compilation crushed again with same log. It's not clear for me what the errors occure. Directory "include" existed. When linking MKL breafly,
> the same. And there was any unsuccessful cases of configuration, with internal libs too.
> 
> QE was builded only in configuration: LAPACK_LIBS  = -latllapack, SCALAPACK_LIBS = , FFT_LIBS  =  -lfftw3,  MPI_LIBS  = 
> -L/usr/lib/mpi/mpi-mpich2-1_5-x86_64/usr/lib64/ , but it works poorly.  Examples go normally on -np 2..8, but jobs (scf, md, relax that start
> normally on local)  always fail due to SEGFAULT. tryed -np 2..128.
>  
> So I ask for advice, has anyone dealt sucesfully with QE on cluster with GFORTRAN? I suspect that gfortran+mkl+QE is unlivable, who knows some
> appropriate configurations?
> 
> PS. Sysadmin's advices didn't help that's why I'm here)
> 
> Sincerely,
> Eugenia W.
> 
> Research engineer
> NRNU, Moscow.
> 
>