[QE-users] [QE-GPU] Problems with error addusdens_gpu : error # 1 cannot allocate aux2_d

[Nelson Andrés Naveas Ríos]

Hi everyone,
I am experimenting with GPUs using qe-7.0 and 7.1. Both versions have been compiled with Nvidia HPC SDK 22.11-CUDA 11.8 and Nvidia HPC SDK 22.9-CUDA 11.7. The workstation is comprised of a single node that houses a 32-core AMD CPU (Ubuntu server 22.04), 64 GB of memory, and x4 Nvidia P100 GPU (a total of 64 GB RAM).
The problem occurs when using Cuda-aware MPI in a relax job of a surface with vacuum and running mpirun -np 8 pw.x -nk 4 -nd 1  -inp input > output.

 The following CRASH message arises:
       task #         3
     from  addusdens_gpu : error #         1
      cannot allocate aux2_d

I have read in the forum that this generally means you are out of GPU memory. However, when I see the output file (and from my experience with this system), the estimated total dynamical RAM is much lower than the 64 GB RAM from the 4xP100 GPUs:

     Estimated max dynamical RAM per process >       3.47 GB
     Estimated total dynamical RAM >      24.30 GB

As I indicated above, this issue arises when using Cuda-aware MPI; without this option, the job is done normally.

Is this CRASH related to the amount of GPU memory? or is there something I am missing in the compilation process? Any advice?

I attached the input, output, and CRASH files, along with the make.inc in the following link:
[https://res-geo.cdn.office.net/assets/mail/file-icon/png/folder_16x16.png]files-qe<https://dauam-my.sharepoint.com/:f:/g/personal/nelson_naveas_estudiante_uam_es/EijZWGjLWQ9Ct8eQO2tGpqcBw-m0s_JKK3R-Cf8LVMIy0g?e=5ycU1G>

Thank you for your time and consideration in advance!
Best regards

Nelson Naveas
Ph.D. student
Applied Physics Department
Universidad Autónoma de Madrid
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20221229/fce0f7a3/attachment.html>