[Pw_forum] [QE-GPU] Maxwell architecture

[Rolly Ng]

Hello Gunnar,

I would recommend Kepler cards (K80 or K40) or even Fermi cards (m2090 or c2075) for QE.

If you are limited by budget, then it would be worth to try the Titan Z and Titan Black which performs decently on QE. I found that 2x Titan Z can double the scf speed comparing to 4x c2075. I have QE v5.3.0, cuda-6.5 and intel PSXE2015 installed on a OpenSUSE 13.2, it works fine so far.

Please be aware of the environmental variables need to be set persistently.

Regards,
Rolly

-----Original Message-----
From: pw_forum-bounces at pwscf.org [mailto:pw_forum-bounces at pwscf.org] On Behalf Of Gunnar Palsson
Sent: 2016年5月11日 22:16
To: pw_forum at pwscf.org
Subject: [Pw_forum] [QE-GPU] Maxwell architecture

Dear all,

We have 2x NVIDIA QUADRO M5000 8 GB and 2x INTEL XEON E5-2699-V3 16 cores running on ubuntu 16.04. I have managed to install the binary nvidia driver, the NVIDIA CUDA toolkit 7.5 and compiled quantum espresso 5.4.0 successfully with intel MKL. I seem to have also been able to compile the QE-GPU version but when I try to run it, it gives the following error message:

***WARNING: unbalanced configuration (1 MPI per node, 2 GPUs per node)
*** ERROR *** something went wrong inside query_gpu_specs! (rank 0)

The configuration was:

export LIBDIRS=“/opt/intel/lib/intel64 /opt/intel/mkl/lib/intel64"
make -f Makefile.gpu distclean
cd GPU
./configure —enable-cuda —enable-parallel —with-phigemm —without-magma —with-gpu-arch=sm_35 cd ..
make -f Makefile.gpu pw-gpu
cd GPU/PW
./pw-gpu.x

I realized that the sm_35 is for the previous generation of cards, so I manually edited the make.sys and changed it to sm_53.
Recompiling with compute_50, sm_50, compute_52, sm_52 or compute_53, sm_53 did not make a difference.

I also manually edited the make.sys and the phigemm.inc and added:
-I/opt/intel/mkl/include -I/opt/intel/mkl/include/intel64/lp64 to IFLAGS

I had to add
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CC) -Xcompiler -fPIC $(COMMON_FLAGS)

to make.sys to avoid a memcpy error during compilation. I also tried without phigemm with no effect.

My question is: Is there a way to compile QE-GPU with the Maxwell architecture and if so how? I read on the forum that unfortunately the Maxwell architecture does not do double precision very well. Is it a prohibitive loss of precision if one restricts the calculations to single precision? I’m really interested in seeing how well these graphics cards work together with the CPUs.

Best regards and thanks in advance,
Gunnar Palsson
_______________________________________________
Pw_forum mailing list
Pw_forum at pwscf.org
http://pwscf.org/mailman/listinfo/pw_forum