[Pw_forum] [QE-GPU] Maxwell architecture

[Gunnar Palsson]

Dear all,

We have 2x NVIDIA QUADRO M5000 8 GB and 2x INTEL XEON E5-2699-V3 16 cores running on ubuntu 16.04. I have managed to install the binary nvidia driver, the NVIDIA CUDA toolkit 7.5 and compiled quantum espresso 5.4.0 successfully with intel MKL. I seem to have also been able to compile the QE-GPU version but when I try to run it, it gives the following error message:

***WARNING: unbalanced configuration (1 MPI per node, 2 GPUs per node)
*** ERROR *** something went wrong inside query_gpu_specs! (rank 0)

The configuration was:

export LIBDIRS=“/opt/intel/lib/intel64 /opt/intel/mkl/lib/intel64"
make -f Makefile.gpu distclean
cd GPU
./configure —enable-cuda —enable-parallel —with-phigemm —without-magma —with-gpu-arch=sm_35
cd ..
make -f Makefile.gpu pw-gpu
cd GPU/PW
./pw-gpu.x

I realized that the sm_35 is for the previous generation of cards, so I manually edited the make.sys and changed it to sm_53.
Recompiling with compute_50, sm_50, compute_52, sm_52 or compute_53, sm_53 did not make a difference.

I also manually edited the make.sys and the phigemm.inc and added:
-I/opt/intel/mkl/include -I/opt/intel/mkl/include/intel64/lp64 to IFLAGS

I had to add 
NVCCFLAGS += -D_FORCE_INLINES -ccbin=$(CC) -Xcompiler -fPIC $(COMMON_FLAGS)

to make.sys to avoid a memcpy error during compilation. I also tried without phigemm with no effect.

My question is: Is there a way to compile QE-GPU with the Maxwell architecture and if so how? I read on the forum that unfortunately the Maxwell architecture does not do double precision very well. Is it a prohibitive loss of precision if one restricts the calculations to single precision? I’m really interested in seeing how well these graphics cards work together with the CPUs.

Best regards and thanks in advance,
Gunnar Palsson