<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div>Hi Andrew,<div class=""><br class=""></div><div class="">This has happened before to me also, and it seems to happen because the login nodes for the XC40 have a slightly different environment than the compute nodes. The configure script is a little smarter on the MOM nodes it seems. I usually do the compilation in an interactive job on the debug (salloc -p debugq —nodes=1). Running a script on the MOM nodes is fine but make sure you use "aprun -n 1 -N 1 make" for the actual make step lest you upset your sysadmin. :o)</div><div class=""><br class=""></div><div class="">In the GNU environment, if you leave LAPACK_LIBS, SCALAPACK_LIBS, BLAS_LIBS, all that stuff (FFTs as well) blank and make sure they’re external, you will be fine. The Cray LibSci library is part of the “ftn” default linking environment and has all the correct symbols.</div><div class=""><br class=""></div><div class="">I agree, the intel environment is a bit harder if you want MKL. You could just use the intel environment with the Cray libsci if you want the extra optimisations from ifort. Both MKL and Cray use an FFTW implementation anyway for Quantum Espresso anyway don’t they? No real benefit to MKL unless you’re after 10%-level optimisations which seems like a waste of time unless you’re actually one of the HPC technicians who run the machine.</div><div class=""><br class=""></div><div class="">Best regards,</div><div class=""><br class=""></div><div class="">Kane</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 11 Nov 2015, at 21:48, Andrew Downs <<a href="mailto:andrew.downs@mail.com" class="">andrew.downs@mail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class=""><div style="font-family: Verdana;font-size: 12.0px;" class=""><div class="">
<div class="">Kane,</div>
<div class=""> </div>
<div class="">Thank you, your suggestions worked, although I think I had to go a step further and get into make.sys and comment out the line where it found the intel BLAS libraries and point it to the internal BLAS.</div>
<div class="">Something about the intel MKL trips up compilation for us, my other researcher isn't able to get it to work either and he's got a good 20 years of experience on me.</div>
<div class=""> </div>
<div class="">-Andrew</div>
<div class="">
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<div style="margin:0 0 10px 0;" class=""><b class="">Sent:</b> Monday, November 09, 2015 at 10:23 PM<br class="">
<b class="">From:</b> "Kane O'Donnell" <<a href="mailto:kane.odonnell@gmail.com" class="">kane.odonnell@gmail.com</a>><br class="">
<b class="">To:</b> "PWSCF Forum" <<a href="mailto:pw_forum@pwscf.org" class="">pw_forum@pwscf.org</a>><br class="">
<b class="">Subject:</b> Re: [Pw_forum] compile attempt with external BLAS and intel MKL (espresso-5.2.1)</div>
<div name="quoted-content" class="">
<div class="">
<div class=""> </div>
Hi Andrew,
<div class=""> </div>
<div class="">
<div class="">Are you trying to just build espresso 5.2.1, or are you trying to build it using some custom linear algebra libraries? On a Cray you’re encouraged strongly *not* to stuff around with custom BLAS/FFT/scalapack/etc. The programming environments are set up so you don’t need to do that, and for example on our XC40 you can build SVN espresso just with:</div>
<div class=""> </div>
<div class="">module swap PrgEnv-cray PrgEnv-gnu</div>
<div class="">./configure —enable-parallel —with-scalapack=yes FC=ftn CC=cc </div>
<div class="">make all</div>
<div class=""> </div>
<div class="">You can do the same for PrgEnv-intel (it’s exactly the same configure line) but it’s still wise to use the MKL link line advisor to change make.sys as necessary - you can copy and paste directly from the link line advisor as Cray set $MKLROOT properly when you load PrgEnv-intel.</div>
<div class=""> </div>
<div class="">Our cluster guys made a wiki entry about QE on the XC40 using the intel suite here:</div>
<div class=""> </div>
<div class=""><a href="https://portal.ivec.org/docs/Supercomputers/Magnus/Quantum_Espresso" target="_blank" class="">https://portal.ivec.org/docs/Supercomputers/Magnus/Quantum_Espresso</a></div>
<div class=""> </div>
<div class="">It’s not quite the same as the link line advisor but it works (except XSpectra, you need the gnu environment to get that to work on our XC40 for some reason). </div>
<div class=""> </div>
<div class="">Kane</div>
<div class=""> </div>
<div class=""> </div>
<div class="">
<div class="">
<blockquote class="">
<div class="">On 10 Nov 2015, at 01:10, Andrew Downs <<a href="x-msg://34/andrew.downs@mail.com" target="_parent" class="">andrew.downs@mail.com</a>> wrote:</div>
<div class="">
<div class="">
<div style="font-family: Verdana;font-size: 12.0px;" class="">
<div class="">Hello,</div>
<div class=""> </div>
<div class="">I've been able to jump a few hurdles that have come up in my compile process, but this one has me stumped.</div>
<div class="">I'm compiling on a Cray XC-40, and another researcher here was able to build 5.1.1 with out BLAS and MKL libraries.</div>
<div class="">He and I assumed 5.2.1 should also work under the same circumstances, but this may turn out not to be the case.</div>
<div class=""> </div>
<div class="">my configure line is as follows:</div>
<div class=""><br class="">
./configure ARCH=x86_64 MPIF90=ftn FFLAGS="-O3" CC=icc CFLAGS=-O3 --prefix=/usr/cta/unsupported/qe/espresso-5.2.1</div>
<div class=""> </div>
<div class="">relevant part of make.sys:</div>
<div class=""> </div>
<div class="">
<div class="">LDFLAGS=--static -I/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/include/ -I/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/include/intel64/lp64/<br class="">
BLAS_LIBS=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_sequential.a -Wl,--end-group /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a -lpthread -lm<br class="">
LAPACK_LIBS=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_core.a<br class="">
SCALAPACK_LIBS=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group<br class="">
FFT_LIBS=/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_cdft_core.a</div>
<div class=""> </div>
</div>
<div class=""> </div>
<div class="">make all output:</div>
<div class="">ftn --static -I/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/include/ -I/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/include/intel64/lp64/ -o pw.x \<br class="">
pwscf.o libpw.a ../../Modules/libqemod.a ../../flib/ptools.a ../../flib/flib.a ../../clib/clib.a ../../iotk/src/libiotk.a /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_scalapack_lp64.a -Wl,--start-group /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_intel_lp64.a /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_core.a /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_cdft_core.a /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_sequential.a -Wl,--end-group /opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a -lpthread -lm<br class="">
/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_core.a(mkl_memory.o): In function `mkl_serv_set_memory_limit':<br class="">
../../../../serv/kernel/mkl_memory.c:(.text+0x521): warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking<br class="">
/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a(dgesd2d_.o): In function `dgesd2d_':<br class="">
../../../../scalapack/BLACS/SRC/MPI/dgesd2d_.c:(.text+0x92): undefined reference to `mkl_serv_get_mpi_wrappers'<br class="">
/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a(dgesd2d_.o): In function `dsendrecv_':<br class="">
../../../../scalapack/BLACS/SRC/MPI/dgesd2d_.c:(.text+0x142): undefined reference to `mkl_serv_get_mpi_wrappers'<br class="">
../../../../scalapack/BLACS/SRC/MPI/dgesd2d_.c:(.text+0x162): undefined reference to `mkl_serv_get_mpi_wrappers'<br class="">
../../../../scalapack/BLACS/SRC/MPI/dgesd2d_.c:(.text+0x1aa): undefined reference to `mkl_serv_get_mpi_wrappers'<br class="">
../../../../scalapack/BLACS/SRC/MPI/dgesd2d_.c:(.text+0x1da): undefined reference to `mkl_serv_get_mpi_wrappers'<br class="">
/opt/intel/compilers_and_libraries_2016.0.109/linux/mkl/lib/intel64/libmkl_blacs_intelmpi_lp64.a(dgesd2d_.o):../../../../scalapack/BLACS/SRC/MPI/dgesd2d_.c:(.text+0x1ea): more undefined references to `mkl_serv_get_mpi_wrappers' follow<br class="">
make[2]: *** [pw.x] Error 1<br class="">
make[2]: Leaving directory `/p/home/adowns/espresso-5.2.1/PW/src'<br class="">
make[1]: *** [pw] Error 1<br class="">
make[1]: Leaving directory `/p/home/adowns/espresso-5.2.1/PW'<br class="">
make: *** [pw] Error 1<br class="">
</div>
</div>
</div>
_______________________________________________<br class="">
Pw_forum mailing list<br class="">
<a href="x-msg://34/Pw_forum@pwscf.org" target="_parent" class="">Pw_forum@pwscf.org</a><br class="">
<a href="http://pwscf.org/mailman/listinfo/pw_forum" target="_blank" class="">http://pwscf.org/mailman/listinfo/pw_forum</a></div>
</blockquote>
</div>
</div>
</div>
_______________________________________________ Pw_forum mailing list <a href="mailto:Pw_forum@pwscf.org" class="">Pw_forum@pwscf.org</a> <a href="http://pwscf.org/mailman/listinfo/pw_forum" target="_blank" class="">http://pwscf.org/mailman/listinfo/pw_forum</a></div>
</div>
</div>
</div>
</div></div></div>
_______________________________________________<br class="">Pw_forum mailing list<br class=""><a href="mailto:Pw_forum@pwscf.org" class="">Pw_forum@pwscf.org</a><br class="">http://pwscf.org/mailman/listinfo/pw_forum</div></blockquote></div><br class=""></div></body></html>