<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof"><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="ContentPasted0">Hi everyone,
<div class="FluidPluginCopy ContentPasted0">I am experimenting with GPUs using qe-7.0 and 7.1. Both versions have been compiled with Nvidia HPC SDK 22.11-CUDA 11.8 and Nvidia HPC SDK 22.9-CUDA 11.7. The workstation is comprised of a single node that houses
a 32-core AMD CPU (Ubuntu server 22.04), 64 GB of memory, and x4 Nvidia P100 GPU (a total of 64 GB RAM).</div>
<div class="FluidPluginCopy ContentPasted0">The problem occurs when using Cuda-aware MPI in a relax job of a surface with vacuum and running mpirun -np 8 pw.x -nk 4 -nd 1 -inp input > output.</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0"> The following CRASH message arises:</div>
<div class="FluidPluginCopy ContentPasted0"> task # 3</div>
<div class="FluidPluginCopy ContentPasted0"> from addusdens_gpu : error # 1</div>
<div class="FluidPluginCopy ContentPasted0"> cannot allocate aux2_d</div>
<div class="FluidPluginCopy ContentPasted0"> </div>
<div class="FluidPluginCopy ContentPasted0">I have read in the forum that this generally means you are out of GPU memory. However, when I see the output file (and from my experience with this system), the estimated total dynamical RAM is much lower than the
64 GB RAM from the 4xP100 GPUs:</div>
<div class="FluidPluginCopy ContentPasted0"> </div>
<div class="FluidPluginCopy ContentPasted0"> Estimated max dynamical RAM per process > 3.47 GB</div>
<div class="FluidPluginCopy ContentPasted0"> Estimated total dynamical RAM > 24.30 GB</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">As I indicated above, this issue arises when using Cuda-aware MPI; without this option, the job is done normally.</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">Is this CRASH related to the amount of GPU memory? or is there something I am missing in the compilation process? Any advice?</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">I attached the input, output, and CRASH files, along with the make.inc in the following link:</div>
<div class="FluidPluginCopy ContentPasted0"><span><a href="https://dauam-my.sharepoint.com/:f:/g/personal/nelson_naveas_estudiante_uam_es/EijZWGjLWQ9Ct8eQO2tGpqcBw-m0s_JKK3R-Cf8LVMIy0g?e=5ycU1G" rel="noopener noreferrer" contenteditable="false" id="OLK_Beautified_OWAb2e8c76a-7858-d81f-68b6-c444cfb865c4" class="OWAAutoLink eScj0 WSYlv" style="padding: 0px 1px; border-radius: 2px; user-select: all; background-color: rgb(243, 242, 241);" data-ogsc="" data-loopstyle="undefined"><img alt="" role="presentation" style="width: 16px; height: 16px; vertical-align: middle; padding: 1px 2px 2px 0px;" src="https://res-geo.cdn.office.net/assets/mail/file-icon/png/folder_16x16.png">files-qe</a></span><br>
</div>
<div class="FluidPluginCopy ContentPasted0"><br>
</div>
<div class="FluidPluginCopy ContentPasted0">Thank you for your time and consideration in advance!</div>
<div class="FluidPluginCopy ContentPasted0">Best regards</div>
<div class="FluidPluginCopy"><br class="ContentPasted0">
</div>
<div class="FluidPluginCopy ContentPasted0">Nelson Naveas </div>
<div class="FluidPluginCopy ContentPasted0">Ph.D. student</div>
<div class="FluidPluginCopy ContentPasted0">Applied Physics Department </div>
Universidad Autónoma de Madrid <br>
</span></div>
</body>
</html>