[Pw_forum] compile attempt with external BLAS and intel MKL (espresso-5.2.1)

[Lorenzo Paulatto]

On Monday, November 09, 2015 06:10:51 PM Andrew Downs wrote:
>  
> LDFLAGS=--static
> -I/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/include/ 
> -I/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/include/intel64/l
> p64/
> BLAS_LIBS=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel
> 64/libmkl_sequential.a -Wl,--end-group
> /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_
> blacs_intelmpi_lp64.a -lpthread -lm
> LAPACK_LIBS=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/int
> el64/libmkl_intel_lp64.a
> /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_
> core.a
> SCALAPACK_LIBS=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/
> intel64/libmkl_scalapack_lp64.a -Wl,--start-group
> FFT_LIBS=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel6
> 4/libmkl_cdft_core.a 
>  


Hello,
this looks funny to me, because you have -Wl,--start-group in one variable and 
-Wl,--end-group in another, and it does no look like the two got concatenated 
in a meaningful way during compilation.

If you got the line from the intel mkl link lien advisor helper, then you do 
not have to cut the line and put it here and there, it is better to put it in 
BLAS_LIBS and leave LAPACK_LIBS and SCALAPACK_LIBS empty. 

Also, your FFTW line does not seem to be correct, if you want to use MKL FFT 
you have to use the proper interface. If your system system admin has already 
compiled it, you can probably grab it from the /whatever/mkl/lib/intel64, it 
is called libfftw3xf_intel.a. If it is not already compiled, you can copy the 
directory /whatever/mkl/interfaces/fftw3xf/ to your home, move to it and 
compile the library with
 make intel64

Furthermore, there is no point in putting all the -I/stuff/ in LDFLAGS: the 
include directories go to IFLAGS (they are passed to the compiler, no to the 
linker).

Finally, you are using the parallel MKL with IntelMPI, i.e. the version of MKL 
that interfaces with the the Intel implementation of MPI libraries. Are you 
sure your MPI wrapper uses the same implementation? (Answer: no, it would be 
called mpiifort). From some googling around, it looks like Cray defaults to 
MPICH2.

So, what you have to do is ditch the wrapper and link MPI manually. In order 
to do that you go to https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor and enter all your informations: 
MKL 11.3
Linux
None
Intel Fortran
Intel 64
Static
LP64
Sequential
[x] Scalapack
MPICH2
And you copy the line:
 ${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group ${MKLROOT}/
lib/intel64/libmkl_intel_lp64.a ${MKLROOT}/lib/intel64/libmkl_core.a $
{MKLROOT}/lib/intel64/libmkl_sequential.a -Wl,--end-group ${MKLROOT}/lib/
intel64/libmkl_blacs_intelmpi_lp64.a -lpthread -lm


Take care to READ THE NOTES! i.e. be sure that MKLROOT variable is define, and 
eventually run
  source /whatever/mkl/bin/mklvars.sh intel64


Then after configuring you clean up BLAS_LIBS and LAPACK_LIBS and put the Link 
Advisor line in SCALAPACK_LIBS.




The problem is that even after fixing everything up to here it may still not 
work, because the ftn wrapper links MPI after MKL, while MKL scalapack 
requires MPI. ANYWAY, WHAT'S ABOVE BEFORE GOING FURTHER, as it usually works.

If it does not work, because you get undefined references to mpi stupp: type 
"ftn -show" to get the MPI linking line 

Than run configure as normal, just to get a canvas make.sys, but then open 
with a text editor and:
1. Put all the -I from the Link Advisor and from "ftn -show" in the IFLAGS
2. clean up BLAS_LIBS and LAPACK_LIBS
3. put the line from Link Advisor in SCALAPACK_LIBS
4. add all the libraries from "ftn -show" at the end of LIBS. 
4b. to be extra safe you can add these libraries between --start-group and --
end-group in SCALAPACK_LIBS, but this is usually not necessary.
5. change MPIF90 from ftn to ifort
6. put the fftw3 library you have compiled in FFT_LIBS (ditch anything that 
could be already there), eventually change -D__FFTW with -D__FFTW3 in DFLAGS. 
In my case this becomes /home/whatever/fftw3xf/libfftw3xf_intel.a
7. "make pw.x" and hope you did not miss anything



HTH

-- 
Dr. Lorenzo Paulatto
IdR @ IMPMC -- CNRS & Université Paris 6
+33 (0)1 44 275 084 / skype: paulatz
http://www.impmc.upmc.fr/~paulatto/
23-24/4é16 Boîte courrier 115, 
4 place Jussieu 75252 Paris Cédex 05