<div dir="ltr">Hi Denis,<div> I was having similar troubles but was able to get it to compile over the weekend, and link to the MKL FFTW3 libraries without building the special interfaces. This was on a cluster composed of Haswell architecture chips. I ran the configure script with ./install/configure BLAS_LIBS="-lmkl_intel_lp64 -lmkl_sequential -lmkl_core" SCALAPACK_LIBS="-lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64" and like you I used the mvapich mpi implementation with the intel 2015 composer suite of Intel compilers. With mvapich you need to tell it to use the<font color="#ff0000"> intelmpi blas</font> libraries rather then the ompi blas libraries, because that is what the configure script will automatically try and use.</div><div><br></div><div> I subsequently manually edited the make.sys file to include external fftw3 libraries, adding the flag -I$(MKLROOT)/include/fftw to IFLAG=-I../ so that it could find the fftw3.h header, and then added FFT_LIBS = -lmkl_intel_lp64 -lmkl_sequential -lmkl_core. This compiled fine without linking to any of the specifically built mkl fftw3 interface wrappers. </div><div><br>I ran a test geometry optimization/sample job using 24 cores because I wasn't sure if that was actually doing anything. The wall time for each identical job was:</div><div> 204 minutes using open-mpi with intel compilers and mkl BLAS/SCALAPACK, </div><div> 158 minutes using mvapich with intel compilers and MKL BLAS/SCALAPACK</div><div> 124 minutes using mvapich with intel compilers and MKL BLAS/SCALAPACK and MKL FFTW3 as linked above.</div><div><br></div><div><br></div><div>I hope this is helpful.</div><div>Best,</div><div>James</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 25, 2016 at 5:17 AM, Denis E. Zavelev <span dir="ltr"><<a href="mailto:metalian@mail.ru" target="_blank">metalian@mail.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>Hello James!<br><br>No, I haven't yet compiled QE.<br>Ddamn, it was really easy to compile QE on my netbook and my desktops (all under linux), but really hard on supercomputers.<br>Filippo promised to help me but he seems to be too busy, I am still waiting an answer from him.<br><br><br><blockquote style="border-left:1px solid #0857a6;margin:10px;padding:0 0 0 10px">
Пятница, 22 января 2016, 15:14 -06:00 от James Johns <<a href="mailto:jjohns@umn.edu" target="_blank">jjohns@umn.edu</a>>:<div><div><br>
<br>
<div>
<div>
<div>
<div><div dir="ltr">Hi Denis,<div> Did you ever get it figured out? I'd be very interested if you did. I too have compiled the MKL wrappers for fftw3, but I can't get QE to recognize the library.</div><div>Best Regards,</div><div>James</div><div><br><div>On Tue, Jan 19, 2016 at 1:04 PM, Paolo Giannozzi <span dir="ltr"><<a href="https://e.mail.ru/compose/?mailto=mailto%3ap.giannozzi@gmail.com" target="_blank">p.giannozzi@gmail.com</a>></span> wrote:<br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>As far as I know, QE uses quite standard MPI calls that work on any reasonable version of MPI. The MPI release you have dates back to 2010. Maybe you should inquire with your system administrator.<br><br></div>Paolo<br></div><div><div><div><br><div>On Tue, Jan 19, 2016 at 6:03 PM, Denis E. Zavelev <span dir="ltr"><<a href="https://e.mail.ru/compose/?mailto=mailto%3ametalian@mail.ru" target="_blank">metalian@mail.ru</a>></span> wrote:<br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><span style="font-family:Arial,Tahoma,Verdana,sans-serif;line-height:20.8px">Hello Filippo!</span><br style="font-family:Arial,Tahoma,Verdana,sans-serif;line-height:20.8px"><br style="font-family:Arial,Tahoma,Verdana,sans-serif;line-height:20.8px"><span style="font-family:Arial,Tahoma,Verdana,sans-serif;line-height:20.8px">Thanks for your quick response.</span><br style="font-family:Arial,Tahoma,Verdana,sans-serif;line-height:20.8px"><span style="font-family:Arial,Tahoma,Verdana,sans-serif;line-height:20.8px">I use MVAPICH-1.2rc1, it's the only available MPI on JSC cluster and I am not allowed to use any other MPI implementations.</span><br style="font-family:Arial,Tahoma,Verdana,sans-serif;line-height:20.8px"><span style="font-family:Arial,Tahoma,Verdana,sans-serif;line-height:20.8px"></span>I've sent you the files you asked.<br><br><br><blockquote style="border-left:1px solid #0857a6;margin:10px;padding:0 0 0 10px">
Среда, 20 января 2016, 0:45 +08:00 от Filippo SPIGA <<a href="https://e.mail.ru/compose/?mailto=mailto%3afilippo.spiga@quantum%2despresso.org" target="_blank">filippo.spiga@quantum-espresso.org</a>>:<div><div><br>
<br>
<div>
<div>
<div>
<div>Dear Denis,<br>
<br>
are you using OpenMPI or Intel MPI?<br>
<br>
If Open MPI, try also <br>
<br>
./configure MPIF90=mpiifort ...<br>
<br>
<br>
If Intel MPI (I assume this is your case), try instead <br>
<br>
./configure MPIF90=mpiifort ... --with-scalapack=intel<br>
<br>
<br>
In the meanwhile please send me personally (not to the mailing-list!) to my email the "install/config.log" file and your make.sys.<br>
<br>
Regards<br>
<br>
--<br>
Mr. Filippo SPIGA, M.Sc.<br>
Quantum ESPRESSO Foundation<br>
<a href="http://www.quantum-espresso.org" target="_blank">http://www.quantum-espresso.org</a> ~ skype: filippo.spiga<br>
<br>
*****<br>
Disclaimer: "Please note this message and any attachments are CONFIDENTIAL and may be privileged or otherwise protected from disclosure. The contents are not to be disclosed to anyone other than the addressee. Unauthorized recipients are requested to preserve this confidentiality and to advise the sender immediately of any error in transmission."<br>
<br>
On Jan 20, 2016, at 12:20 AM, Denis E. Zavelev <<a href="https://e.mail.ru/compose?To=metalian@mail.ru" target="_blank">metalian@mail.ru</a>> wrote:<br>
> Hello!<br>
> <br>
> I am trying to compile QE on JSC RAS cluster. As I have user permissions, I can install any programs only locally.<br>
> Cluster works under Linux. We have intel compilers (mpif90, icc, ifort) and MKL installed on cluster. No FFTW libraries are installed though even MKL ones (I have to build them locally). <br>
> <br>
> I've downloaded espresso 5.3.0 from the site. <br>
> Configure script finished successfully. But then I got 2 warnings and subsequent error message when compiling internal FFT.<br>
> <br>
> I decided to use some other FFT libs.<br>
> So I've downloaded FFTW3 from its site, successfully built, tested and installed it. But the same error message. <br>
> <br>
> Libraries found by configure script:<br>
> BLAS_LIBS= -lmkl_intel_lp64 -lmkl_sequential -lmkl_core<br>
> LAPACK_LIBS=<br>
> SCALAPACK_LIBS=-lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64<br>
> FFT_LIBS= -lfftw3 <br>
> <br>
> Here's the output:<br>
> <br>
> ============================<br>
> bash-3.2$ make pw<br>
> make: Warning: File `make.sys' has modification time 0.47 s in the future<br>
> test -d bin || mkdir bin<br>
> ( cd FFTXlib ; make TLDEPS= all || exit 1 )<br>
> make[1]: Entering directory `/nethome/metalian/espresso/espresso-5.3.0/FFTXlib'<br>
> make[1]: Warning: File `../make.sys' has modification time 0.46 s in the future<br>
> mpif90 -O2 -assume byterecl -g -traceback -par-report0 -vec-report0 -nomodule -fpp -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -I../include -I../iotk/src -I. -c fft_types.f90<br>
> mpif90 -O2 -assume byterecl -g -traceback -par-report0 -vec-report0 -nomodule -fpp -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -I../include -I../iotk/src -I. -c scatter_mod.f90<br>
> icc -O3 -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -I../include -c fftw.c<br>
> fftw.c(27449): warning #188: enumerated type mixed with another type<br>
> EXPECT_INT(dir);<br>
> ^<br>
> <br>
> fftw.c(27450): warning #188: enumerated type mixed with another type<br>
> EXPECT_INT(type);<br>
> ^<br>
> <br>
> mpif90 -O2 -assume byterecl -g -traceback -par-report0 -vec-report0 -nomodule -fpp -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -I../include -I../iotk/src -I. -c fft_scalar.f90<br>
> mpif90 -O2 -assume byterecl -g -traceback -par-report0 -vec-report0 -nomodule -fpp -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -I../include -I../iotk/src -I. -c fft_parallel.f90<br>
> mpif90 -O2 -assume byterecl -g -traceback -par-report0 -vec-report0 -nomodule -fpp -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -I../include -I../iotk/src -I. -c fft_smallbox.f90<br>
> mpif90 -O2 -assume byterecl -g -traceback -par-report0 -vec-report0 -nomodule -fpp -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -I../include -I../iotk/src -I. -c fft_interfaces.f90<br>
> mpif90 -O2 -assume byterecl -g -traceback -par-report0 -vec-report0 -nomodule -fpp -D__INTEL -D__FFTW3 -D__MPI -D__PARA -D__SCALAPACK -I../include -I../iotk/src -I. -c stick_base.f90<br>
> stick_base.f90(169): error #6404: This name does not have a type, and must have an explicit type. [MPI_IN_PLACE]<br>
> CALL MPI_ALLREDUCE(MPI_IN_PLACE, st, SIZE(st), MPI_INTEGER, MPI_SUM, comm, ierr)<br>
> -----------------------------^<br>
> compilation aborted for stick_base.f90 (code 1)<br>
> make[1]: *** [stick_base.o] Error 1<br>
> make[1]: Leaving directory `/nethome/metalian/espresso/espresso-5.3.0/FFTXlib'<br>
> make: *** [libfft] Error 1<br>
> ============================<br>
> <br>
> I've also tries espresso 5.2.1. Configure works the same way, but compilation also fails though not so fast, it ends on the following:<br>
> <br>
> fft_scalar.f90(69): #error: can't find include file: fftw3.f<br>
> make[1]: *** [fft_scalar.o] Error 1<br>
> make[1]: Leaving directory `/nethome/metalian/espresso/espresso-5.2.1/Modules'<br>
> make: *** [mods] Error 1<br>
> <br>
> This is strange because<br>
> -sh-3.2$ ls /nethome/metalian/soft/include/<br>
> fftw3.f fftw3.f03 fftw3.h fftw3l.f03 fftw3q.f03<br>
> <br>
> -sh-3.2$ export<br>
> export C_INCLUDE_PATH="/nethome/metalian/soft/include"<br>
> export INCLUDE="/nethome/metalian/soft/include:/opt/intel/composerxe-2011.3.174/mkl/include:/opt/intel/composerxe-2011.3.174/mkl/include"<br>
> <br>
> [I've mentioned only environment variables dealing with including]<br>
> <br>
> <br>
> So what am I doing wrong?<br>
> Are there any experienced users of Quantum Expresso on JSC cluster?<br>
> <br>
> <br>
> <br>
> Best regards,<br>
> Denis E. Zavelev<br>
> <br>
> _______________________________________________<br>
> Pw_forum mailing list<br>
> <a href="https://e.mail.ru/compose?To=Pw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
> <a href="http://pwscf.org/mailman/listinfo/pw_forum" target="_blank">http://pwscf.org/mailman/listinfo/pw_forum</a><br>
<br>
</div>
</div>
</div>
</div>
</div></div></blockquote><div><div>
<br>
<br>Best regards,<br>Denis E. Zavelev<br></div></div></div>
<br>_______________________________________________<br>
Pw_forum mailing list<br>
<a href="https://e.mail.ru/compose/?mailto=mailto%3aPw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/listinfo/pw_forum</a><br></blockquote></div><br><br clear="all"><br></div></div><span><font color="#888888">-- <br><div><div dir="ltr"><div><div dir="ltr"><div>Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>Phone <a value="+390432558216"><span>+39-0432-558216</span></a>, fax <a value="+390432558222"><span>+39-0432-558222</span></a><br><br></div></div></div></div></div>
</font></span></div>
<br>_______________________________________________<br>
Pw_forum mailing list<br>
<a href="https://e.mail.ru/compose/?mailto=mailto%3aPw_forum@pwscf.org" target="_blank">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/listinfo/pw_forum</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div>James E. Johns</div><div>Assistant Professor</div><div>Department of Chemistry</div><div>University of Minnesota</div><div>207 Pleasant St.</div><div>Minneapolis, MN 55455</div><div><a href="http://www.chem.umn.edu/groups/johns/index.html" target="_blank">http://www.chem.umn.edu/groups/johns/index.html</a><br></div></div></div>
</div></div>
</div>
</div>
</div>
</div>
</div></div></blockquote>
<br>
<br><br></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div><div dir="ltr"><div>James E. Johns</div><div>Assistant Professor</div><div>Department of Chemistry</div><div>University of Minnesota</div><div>207 Pleasant St.</div><div>Minneapolis, MN 55455</div><div><a href="http://www.chem.umn.edu/groups/johns/index.html" target="_blank">http://www.chem.umn.edu/groups/johns/index.html</a><br></div></div></div>
</div></div>