[Pw_forum] Compiling with intel fortran, MKL and MKL FFT wrappers

[Serge Nakhmanson]

Lorenzo, Axel,

Thank you for your comments. I expected the serial lib
to be "it" but decided to check in with the forum just
to be sure. I have not tested the new "MKL FTT" binary
extensively yet but I will. As I understand, I should not
expect a tremendous performance improvement over plain
vanilla fftw2 lib, however, if I see something interesting
I will drop a line here.


What is given below is a comment to answer Vega's questions.
It could be skipped unless one's interested in the nitty-gritty
details and/or wants to find and expose flaws in my operating
procedures.

 > could you tell me which dir should be added to the LD_LIBRARY_PATH, or which
 > flags should be added after the QE configure command?

Vega,

Here's what you can do, if, against Axel's advice, you
just gotta have an MKL FFT binary in your {espresso}/bin.

I usually do not use configure. Instead I edit make.sys
directly, which seems to be much more convenient. If you
take a look at the end of my original message you will
see how I set all the important flags (including include
and lib paths) to compile everything without problems.
The only thing you have to do is put in your own paths
to MKL instead of my paths.

Regarding the FFT_LIBS flag, by default, my sysadmin placed
all the compiled MKL FFT libs into {mkl}/{number}/lib/64
but you can place them anywhere as long as you provide a
correct path to them in make.sys.

Regarding the LD_LIBRARY_PATH, I usually set it inside my
PBS job script, e.g.,

#
# Set up some compiler-dependent things:
#
if [ $COMPILER = intel ]; then

   # Set up some things to run an Intel-compiler binary:
   MPIRUN=/usr/local/bin/mpiexec
   MPIFLAGS='-verbose -kill'
   export 
LD_LIBRARY_PATH=/usr/local/intel/mkl/10.0.4.023/lib/em64t/:$LD_LIBRARY_PATH

elif [ $COMPILER = pgi64 ]; then

   {do something else here}

else

   echo "Compiler $COMPILER unsupported for parallel execution"
   exit 1

fi

You can then check if all the appropriate libs are found correctly
by doing "ldd your_binary.x" (in the same PBS script, of course)
before you run it:

# Run the job in /<local scratch>/<run>:

DATE=`date`
echo ""
echo "Time at the start of the job is: $DATE"
echo ""

cd $SCRATCH/$RUNDIR

echo ""
echo "ldd pw.x:"
ldd $SCRATCH/$RUNDIR/pw.$ARCH.x
echo ""

echo ""
echo "Running the job:"
echo "$MPIRUN $MPIFLAGS $SCRATCH/$RUNDIR/pw.$ARCH.x $NPOOL < 
$SCRATCH/$RUNDIR/$NAME.$PREFIX.pw.inp > $SCRATCH/$RUNDIR/$NAME.$PREFIX.pw.out"
echo ""
$MPIRUN $MPIFLAGS $SCRATCH/$RUNDIR/pw.$ARCH.x $NPOOL < 
$SCRATCH/$RUNDIR/$NAME.$PREFIX.pw.inp > $SCRATCH/$RUNDIR/$NAME.$PREFIX.pw.out

DATE=`date`
echo ""
echo "Time at the end of the job is: $DATE"
echo "Doing some housekeeping duties..."
echo ""


Here's a sample output from this part of the script that I get on my cluster,
which shows that at least the MKL libraries from BLAS_LIBS and LAPACK_LIBS
are found correctly:

 > Time at the start of the job is: Mon Sep 29 17:37:18 CDT 2008
 >
 >
 > ldd pw.x:
 >         libmkl_intel_lp64.so => 
/usr/local/intel/mkl/10.0.4.023/lib/em64t/libmkl_intel_lp64.so (0x0000002a95557000)
 >         libmkl_sequential.so => 
/usr/local/intel/mkl/10.0.4.023/lib/em64t/libmkl_sequential.so (0x0000002a9586d000)
 >         libmkl_core.so => 
/usr/local/intel/mkl/10.0.4.023/lib/em64t/libmkl_core.so (0x0000002a95a1a000)
 >         libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x000000375f300000)
 >         librt.so.1 => /lib64/tls/librt.so.1 (0x000000375e900000)
 >         libm.so.6 => /lib64/tls/libm.so.6 (0x000000375e300000)
 >         libc.so.6 => /lib64/tls/libc.so.6 (0x000000375de00000)
 >         libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000375eb00000)
 >         libdl.so.2 => /lib64/libdl.so.2 (0x000000375e100000)
 >         /lib64/ld-linux-x86-64.so.2 (0x000000375dc00000)
 >
 >
 > Running the job:
 > /usr/local/bin/mpiexec -verbose -kill /scratch/nakhmanson/PWtest3/pw.test.x 
-npool 2 < /scratch/nakhmanson/PWtest3/C1c.relaxed.pw.inp > /scratch/nakhmanson/PWt
 > est3/C1c.relaxed.pw.out
 >
 > mpiexec: resolve_exe: using absolute path 
"/scratch/nakhmanson/PWtest3/pw.test.x".
 > mpiexec: process_start_event: evt 2 task 0 on node09.cluster.
 > mpiexec: read_p4_master_port: waiting for port from master.
 > mpiexec: read_p4_master_port: got port 33007.
 > mpiexec: process_start_event: evt 4 task 1 on node09.cluster.
 > mpiexec: All 2 tasks (spawn 0) started.
 > mpiexec: wait_tasks: waiting for node09.cluster node09.cluster.
 > mpiexec: process_obit_event: evt 3 task 0 on node09.cluster stat 0.
 > mpiexec: process_obit_event: evt 5 task 1 on node09.cluster stat 0.
 >
 > Time at the end of the job is: Mon Sep 29 17:41:03 CDT 2008
 > Doing some housekeeping duties...

Hope this helps,

Serge

-- 
*********************************************************
  Serge M. Nakhmanson               phone: (630) 252-5205
  Assistant Scientist                 fax: (630) 252-4798
  MSD-212, Rm. C-224
  Argonne National Laboratory
  9700 S. Cass Ave.
  Argonne, IL 60439
*********************************************************