[QE-users] Help on QE + GPU + OpenMP compilation

[Giovanni Cantele]

Hi all,

I’m trying to compile the latest QE version with OpenMP and GPU support. The latest NVIDIA hpc package is in use (nvidia/hpc_sdk/Linux_x86_64/21.3). 4xTesla V100 SXM2 32GB are mounted on the computing node.


There are some points that are not clear to me and I would be grateful if someone might help me in this respect:

i) configure command
export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/21.3/comm_libs/openmpi/openmpi-3.1.5/bin/:/opt/nvidia/hpc_sdk/Linux_x86_64/21.3/compilers/bin/:$PATH
./install/configure FC=nvfortran CC=nvcc --enable-openmp --with-cuda=/usr/local/cuda --with-cuda-cc=70 --with-cuda-runtime=11.2 LIBDIRS=/opt/nvidia/hpc_sdk/Linux_x86_64/21.3/compilers/lib/ /opt/nvidia/hpc_sdk/Linux_x86_64/21.3/math_libs/11.2/targets/x86_64-linux/lib/stubs /opt/nvidia/hpc_sdk/Linux_x86_64/21.3/comm_libs/openmpi/openmpi-3.1.5/lib/ /opt/nvidia/hpc_sdk/Linux_x86_64/21.3/comm_libs/mpi/lib/

I’ve given the needed paths and libraries that should suffice to a proper installation with CUDA, lapack, blas and so on. The following compilation does work. However, I find something strange in
make.inc :
LAPACK_LIBS    = -L/opt/nvidia/hpc_sdk/Linux_x86_64/21.3/comm_libs/openmpi/openmpi-3.1.5/lib/ -llapack  -lblas
The -L…. directory, among those provided through LIBDIRS, is NOT the one containing lapack and blas libraries, so although the subsequent compilation seems to work, the link as above seems to be wrong (it shouldn’t work!). Is that normal? For example, it seems that a library containing dgeem is not found. However, /opt/nvidia/hpc_sdk/Linux_x86_64/21.3/compilers/lib/  contains both libblas.a and liblapack.a .

The install/config.log is attached for further checks.


ii) CUDA support seems to have been correctly included. I tried to run pw.x and it seems to work. Also, using nvidia-smi, I see that the pw.x processes are running on the GPUs. However, while on Marconi100 the output file contains a section like
    GPU acceleration is ACTIVE.


     GPU used by master process:

        Device Number: 0
        Device name: Tesla V100-SXM2-16GB
        Compute capability : 70
        Ratio of single to double precision performance  : 2
        Memory Clock Rate (KHz): 877000
        Memory Bus Width (bits): 4096
        Peak Memory Bandwidth (GB/s): 898.05
, I do not find any message like this in my output. Does it mean that the GPU support has some troubles?

iii) last point, it seems that OpenMP support has not been included. I cannot find any indication of that either in config.log or in make.inc (also attached to this message), despite —enable-openmp was specified to the configure command. I also do not find any information in the output concerning the number of threads. Is there anything else I should specify at compiling / running time ?


I’m very sorry for overburdening you with so many questions, but any help would be very much appreciated.

Thanks,

    Giovanni








-- 

Giovanni Cantele, PhD
CNR-SPIN
c/o Dipartimento di Fisica
Universita' di Napoli "Federico II"
Complesso Universitario M. S. Angelo - Ed. 6
Via Cintia, I-80126, Napoli, Italy
e-mail: giovanni.cantele at spin.cnr.it
Phone: +39 081 676910
Skype contact: giocan74

ResearcherID: http://www.researcherid.com/rid/A-1951-2009
Web page: https://sites.google.com/view/giovanni-cantele/home

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20210415/d2731826/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: application/octet-stream
Size: 41306 bytes
Desc: not available
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20210415/d2731826/attachment.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20210415/d2731826/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.inc
Type: application/octet-stream
Size: 6334 bytes
Desc: not available
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20210415/d2731826/attachment-0001.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20210415/d2731826/attachment-0002.html>