<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The GPU executable can be launched in the same way as the CPU one, but considering this:</div>
<ul style="margin-top: 0px; margin-bottom: 0px;" data-editing-info="{"orderedStyleType":1,"unorderedStyleType":2}">
<li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div class="elementToProof">the number of mpi per node must be the same as the number of GPUs (2 mpi per node in your case). In principle you can try to use more mpi processes per GPU, but it is not recommended;</div>
</li><li style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); list-style-type: "- ";">
<div class="elementToProof">you can enable openMP together with GPU (add --enable-openmp to ./configure) in order to exploit CPU threading in the few places where GPU porting is not present (no more than 8 thread per node, generally doesn't make much difference
though)</div>
</li></ul>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I don't know which scheduler is in use in your system, here is an example of a batch job in slurm launching on 2 nodes with 2 GPUs:</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
------------------------------------------------------------------------------------------------------------<br>
#!/bin/bash <br>
#SBATCH --nodes=2<br>
#SBATCH --ntasks-per-node=2<br>
#SBATCH --cpus-per-task=1<br>
#SBATCH --gres=gpu:2 <br>
#SBATCH --time=00:20:00 <br>
<br>
module purge <br>
module load hpcsdk/24.3</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
export<b> OMP_NUM_THREADS=1</b></div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
mpirun -np 4 /home/q-e/bin/pw.x -nk 1 -nb 1 -input scf.in > scf.out<br>
---------------------------------------------------------------------------------------------------------------</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hope it helps</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Cheers,</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Fabrizio</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mauro Francesco Sgroi <maurofrancesco.sgroi@unito.it><br>
<b>Sent:</b> Friday, August 2, 2024 2:35 PM<br>
<b>To:</b> Fabrizio Ferrari Ruffino <faferrar@sissa.it><br>
<b>Cc:</b> Quantum ESPRESSO users Forum <users@lists.quantum-espresso.org><br>
<b>Subject:</b> Re: [QE-users] Help for compilation with Nvidia HPC SDK</font>
<div> </div>
</div>
<div>
<div dir="ltr">Dear Fabrizio,
<div>thanks a lot for the explanation.</div>
<div>I was unsure about how to proceed and worried not to get the proper performance on the GPU.</div>
<div><br>
</div>
<div>May I ask for help regarding the way of running the code? Where can I find instructions on how to launch the executable?</div>
<div><br>
</div>
<div>For example, how can I control the number of GPUs used and the parallel processes? </div>
<div><br>
</div>
<div>I have 2 GPUs for each node.</div>
<div><br>
</div>
<div>Thanks a lot and best regards,</div>
<div>Mauro Sgroi.<br clear="all">
<div>
<div dir="ltr" class="x_gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000" face="tahoma, sans-serif">___</font><font color="#000000" face="tahoma, sans-serif">____________________</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">Dr.
</span><b style="color:rgb(0,0,0)">Mauro Francesco Sgroi</b><br>
</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font face="tahoma, sans-serif"><font color="#000000"><font>Department of Chemistry
</font></font></font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000" face="tahoma, sans-serif">University of Turin</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000" face="tahoma, sans-serif">Via Quarello 15a</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000" face="tahoma, sans-serif">I-10135 TORINO (Italy)</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">Tel.</font></span></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-8372 </font></span></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-7364</font></span></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">e-mail:
</span><a href="mailto:maurofrancesco.sgroi@unito.it" target="_blank">maurofrancesco.sgroi@unito.it</a><br>
</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font face="tahoma, sans-serif"><font color="#000000"><font>Web: </font></font></font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><a href="http://www.met.unito.it/" rel="noreferrer" target="_blank" style="color:rgb(17,85,204)">www.met.unito.it</a><br>
</p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><a href="http://www.chimica.unito.it" target="_blank" style="font-family:tahoma,sans-serif">www.chimica.unito.it</a><span style="color:rgb(0,0,0); font-family:tahoma,sans-serif"> </span><br>
</p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000"><font><font face="tahoma, sans-serif">Orcid:
<a href="https://orcid.org/0000-0002-0914-4217" target="_blank">https://orcid.org/0000-0002-0914-4217</a></font>
</font></font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000"><font>Webex: </font><a href="https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi" target="_blank">https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi</a> </font></p>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">Il giorno ven 2 ago 2024 alle ore 14:11 Fabrizio Ferrari Ruffino <<a href="mailto:faferrar@sissa.it">faferrar@sissa.it</a>> ha scritto:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg-4131132194137127674">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
there are a few minor FFTXlib calls somewhere in QE which are still on CPU, therefore it is better to have a CPU fft backend enabled too. Whether to use the internal one or FFTW3 should not make much difference, since all the main stuff runs on gpu (therefore
calling cuFFT).<br>
In a CPU run the FFTW3 backend is faster than the internal one, but, as I said, in a GPU run it should be quite irrelevant.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Cheers,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Fabrizio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
CNR IOM</div>
<div id="x_m_-4131132194137127674appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-4131132194137127674divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> users <<a href="mailto:users-bounces@lists.quantum-espresso.org" target="_blank">users-bounces@lists.quantum-espresso.org</a>>
on behalf of Mauro Francesco Sgroi via users <<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
<b>Sent:</b> Friday, August 2, 2024 12:13 PM<br>
<b>To:</b> Quantum ESPRESSO users Forum <<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
<b>Subject:</b> [QE-users] Help for compilation with Nvidia HPC SDK</font>
<div> </div>
</div>
<div>
<div dir="ltr">Dear all,
<div>
<div>I am trying to compile the 7.3.1 version of Quantum Espresso using the last Nvidia HPC SDK (24.7) on Ubuntu 24.04.</div>
<div><br>
</div>
<div>I am configuring as follows:</div>
<div><br>
</div>
<div>export BLAS_LIBS='-L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/math_libs/lib64 -lcublas -lcublasLt -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/compilers/lib -lblas -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/cuda/lib64 -lcudart'<br>
<br>
</div>
<div>export LAPACK_LIBS='-L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/math_libs/lib64 -lcusolver -lcurand -lcublas -lcublasLt -lcusparse -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/compilers/lib -llapack -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/cuda/lib64 -lcudart'<br>
<br>
</div>
<div>export SCALAPACK_LIBS='-/opt/nvidia/hpc_sdk/Linux_x86_64/2024/comm_libs/12.5/openmpi4/openmpi-4.1.5/lib -lscalapack -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/comm_libs/12.5/openmpi4/latest/lib -lmpi -lopen-pal'</div>
<div><br>
</div>
<div>./configure --with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/2024/cuda/12.5 --with-cuda-cc=75 --with-cuda-runtime=12.5 --with-cuda-mpi=yes</div>
<div><br>
</div>
<div>In this way, the internal FFTW library is selected. Should I use the FFTW3 library together with cufft? <br>
</div>
<div><br>
</div>
<div>Can the two libraries work together? Is it normal that the internal FFTW library is used? Or should the cufft library be sufficient?</div>
<div><br>
</div>
<div>Or is it better to use the cufftw library supplied by NVIDIA? </div>
<div><br>
</div>
<div>Can I have some guidance on these aspects?</div>
<div><br>
</div>
<div>Thanks a lot and best regards,</div>
<div>Mauro Sgroi.</div>
<div><br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000" face="tahoma, sans-serif">___</font><font color="#000000" face="tahoma, sans-serif">____________________</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">Dr.
</span><b style="color:rgb(0,0,0)">Mauro Francesco Sgroi</b><br>
</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font face="tahoma, sans-serif"><font color="#000000"><font>Department of Chemistry
</font></font></font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000" face="tahoma, sans-serif">University of Turin</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000" face="tahoma, sans-serif">Via Quarello 15a</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000" face="tahoma, sans-serif">I-10135 TORINO (Italy)</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">Tel.</font></span></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-8372 </font></span></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-7364</font></span></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">e-mail:
</span><a href="mailto:maurofrancesco.sgroi@unito.it" target="_blank">maurofrancesco.sgroi@unito.it</a><br>
</font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font face="tahoma, sans-serif"><font color="#000000"><font>Web: </font></font></font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><a href="http://www.met.unito.it/" rel="noreferrer" target="_blank" style="color:rgb(17,85,204)">www.met.unito.it</a><br>
</p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><a href="http://www.chimica.unito.it" target="_blank" style="font-family:tahoma,sans-serif">www.chimica.unito.it</a><span style="color:rgb(0,0,0); font-family:tahoma,sans-serif"> </span><br>
</p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000"><font><font face="tahoma, sans-serif">Orcid:
<a href="https://orcid.org/0000-0002-0914-4217" target="_blank">https://orcid.org/0000-0002-0914-4217</a></font>
</font></font></p>
<p lang="zxx" style="margin-bottom:0cm; line-height:100%"><font color="#000000"><font>Webex: </font><a href="https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi" target="_blank">https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi</a> </font></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>