<div dir="ltr">Dear Fabrizio,<div>thanks a lot for the help.</div><div>Now my compilation works.</div><div>Best regards,<br></div><div>Mauro Sgroi.</div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">___</font><font color="#000000" face="tahoma, sans-serif">____________________</font></p><p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">Dr.
</span><b style="color:rgb(0,0,0)">Mauro Francesco Sgroi</b><br></font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%">
<font face="tahoma, sans-serif"><font color="#000000"><font>Department
of Chemistry </font></font>
</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%">
<font color="#000000" face="tahoma, sans-serif">University
of Turin</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">Via Quarello 15a</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%">
<font color="#000000" face="tahoma, sans-serif">I-10135
TORINO (Italy)</font></p><p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">Tel.</font></span></p><p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-8372 </font></span></p><p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-7364</font></span></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">e-mail:
</span><a href="mailto:maurofrancesco.sgroi@unito.it" target="_blank">maurofrancesco.sgroi@unito.it</a><br></font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%">
<font face="tahoma, sans-serif"><font color="#000000"><font>Web: </font></font></font></p><p lang="zxx" style="margin-bottom:0cm;line-height:100%"><a href="http://www.met.unito.it/" rel="noreferrer" style="color:rgb(17,85,204)" target="_blank">www.met.unito.it</a><br></p><p lang="zxx" style="margin-bottom:0cm;line-height:100%"><a href="http://www.chimica.unito.it" style="font-family:tahoma,sans-serif" target="_blank">www.chimica.unito.it</a><span style="color:rgb(0,0,0);font-family:tahoma,sans-serif"> </span><br></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%">
<font color="#000000"><font><font face="tahoma, sans-serif">Orcid:
<a href="https://orcid.org/0000-0002-0914-4217" target="_blank">https://orcid.org/0000-0002-0914-4217</a></font>
</font></font>
</p><p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000"><font>Webex: </font><a href="https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi" target="_blank">https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi</a> </font></p></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 2 ago 2024 alle ore 15:13 Fabrizio Ferrari Ruffino <<a href="mailto:faferrar@sissa.it" target="_blank">faferrar@sissa.it</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The GPU executable can be launched in the same way as the CPU one, but considering this:</div>
<ul style="margin-top:0px;margin-bottom:0px">
<li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<div>the number of mpi per node must be the same as the number of GPUs (2 mpi per node in your case). In principle you can try to use more mpi processes per GPU, but it is not recommended;</div>
</li><li style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0);list-style-type:"- "">
<div>you can enable openMP together with GPU (add --enable-openmp to ./configure) in order to exploit CPU threading in the few places where GPU porting is not present (no more than 8 thread per node, generally doesn't make much difference
though)</div>
</li></ul>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
I don't know which scheduler is in use in your system, here is an example of a batch job in slurm launching on 2 nodes with 2 GPUs:</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
------------------------------------------------------------------------------------------------------------<br>
#!/bin/bash <br>
#SBATCH --nodes=2<br>
#SBATCH --ntasks-per-node=2<br>
#SBATCH --cpus-per-task=1<br>
#SBATCH --gres=gpu:2 <br>
#SBATCH --time=00:20:00 <br>
<br>
module purge <br>
module load hpcsdk/24.3</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
export<b> OMP_NUM_THREADS=1</b></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
mpirun -np 4 /home/q-e/bin/pw.x -nk 1 -nb 1 -input <a href="http://scf.in" target="_blank">scf.in</a> > scf.out<br>
---------------------------------------------------------------------------------------------------------------</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hope it helps</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Cheers,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Fabrizio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="m_83718684629268056m_5211245129785990361appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_83718684629268056m_5211245129785990361divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Mauro Francesco Sgroi <<a href="mailto:maurofrancesco.sgroi@unito.it" target="_blank">maurofrancesco.sgroi@unito.it</a>><br>
<b>Sent:</b> Friday, August 2, 2024 2:35 PM<br>
<b>To:</b> Fabrizio Ferrari Ruffino <<a href="mailto:faferrar@sissa.it" target="_blank">faferrar@sissa.it</a>><br>
<b>Cc:</b> Quantum ESPRESSO users Forum <<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
<b>Subject:</b> Re: [QE-users] Help for compilation with Nvidia HPC SDK</font>
<div> </div>
</div>
<div>
<div dir="ltr">Dear Fabrizio,
<div>thanks a lot for the explanation.</div>
<div>I was unsure about how to proceed and worried not to get the proper performance on the GPU.</div>
<div><br>
</div>
<div>May I ask for help regarding the way of running the code? Where can I find instructions on how to launch the executable?</div>
<div><br>
</div>
<div>For example, how can I control the number of GPUs used and the parallel processes? </div>
<div><br>
</div>
<div>I have 2 GPUs for each node.</div>
<div><br>
</div>
<div>Thanks a lot and best regards,</div>
<div>Mauro Sgroi.<br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">___</font><font color="#000000" face="tahoma, sans-serif">____________________</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">Dr.
</span><b style="color:rgb(0,0,0)">Mauro Francesco Sgroi</b><br>
</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><font color="#000000"><font>Department of Chemistry
</font></font></font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">University of Turin</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">Via Quarello 15a</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">I-10135 TORINO (Italy)</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">Tel.</font></span></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-8372 </font></span></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-7364</font></span></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">e-mail:
</span><a href="mailto:maurofrancesco.sgroi@unito.it" target="_blank">maurofrancesco.sgroi@unito.it</a><br>
</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><font color="#000000"><font>Web: </font></font></font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><a href="http://www.met.unito.it/" rel="noreferrer" style="color:rgb(17,85,204)" target="_blank">www.met.unito.it</a><br>
</p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><a href="http://www.chimica.unito.it" style="font-family:tahoma,sans-serif" target="_blank">www.chimica.unito.it</a><span style="color:rgb(0,0,0);font-family:tahoma,sans-serif"> </span><br>
</p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000"><font><font face="tahoma, sans-serif">Orcid:
<a href="https://orcid.org/0000-0002-0914-4217" target="_blank">https://orcid.org/0000-0002-0914-4217</a></font>
</font></font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000"><font>Webex: </font><a href="https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi" target="_blank">https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi</a> </font></p>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div>
<div dir="ltr">Il giorno ven 2 ago 2024 alle ore 14:11 Fabrizio Ferrari Ruffino <<a href="mailto:faferrar@sissa.it" target="_blank">faferrar@sissa.it</a>> ha scritto:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Hi,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
there are a few minor FFTXlib calls somewhere in QE which are still on CPU, therefore it is better to have a CPU fft backend enabled too. Whether to use the internal one or FFTW3 should not make much difference, since all the main stuff runs on gpu (therefore
calling cuFFT).<br>
In a CPU run the FFTW3 backend is faster than the internal one, but, as I said, in a GPU run it should be quite irrelevant.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Cheers,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Fabrizio</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
CNR IOM</div>
<div id="m_83718684629268056m_5211245129785990361x_m_-4131132194137127674appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="m_83718684629268056m_5211245129785990361x_m_-4131132194137127674divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> users <<a href="mailto:users-bounces@lists.quantum-espresso.org" target="_blank">users-bounces@lists.quantum-espresso.org</a>>
on behalf of Mauro Francesco Sgroi via users <<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
<b>Sent:</b> Friday, August 2, 2024 12:13 PM<br>
<b>To:</b> Quantum ESPRESSO users Forum <<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
<b>Subject:</b> [QE-users] Help for compilation with Nvidia HPC SDK</font>
<div> </div>
</div>
<div>
<div dir="ltr">Dear all,
<div>
<div>I am trying to compile the 7.3.1 version of Quantum Espresso using the last Nvidia HPC SDK (24.7) on Ubuntu 24.04.</div>
<div><br>
</div>
<div>I am configuring as follows:</div>
<div><br>
</div>
<div>export BLAS_LIBS='-L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/math_libs/lib64 -lcublas -lcublasLt -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/compilers/lib -lblas -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/cuda/lib64 -lcudart'<br>
<br>
</div>
<div>export LAPACK_LIBS='-L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/math_libs/lib64 -lcusolver -lcurand -lcublas -lcublasLt -lcusparse -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/compilers/lib -llapack -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/cuda/lib64 -lcudart'<br>
<br>
</div>
<div>export SCALAPACK_LIBS='-/opt/nvidia/hpc_sdk/Linux_x86_64/2024/comm_libs/12.5/openmpi4/openmpi-4.1.5/lib -lscalapack -L/opt/nvidia/hpc_sdk/Linux_x86_64/2024/comm_libs/12.5/openmpi4/latest/lib -lmpi -lopen-pal'</div>
<div><br>
</div>
<div>./configure --with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/2024/cuda/12.5 --with-cuda-cc=75 --with-cuda-runtime=12.5 --with-cuda-mpi=yes</div>
<div><br>
</div>
<div>In this way, the internal FFTW library is selected. Should I use the FFTW3 library together with cufft? <br>
</div>
<div><br>
</div>
<div>Can the two libraries work together? Is it normal that the internal FFTW library is used? Or should the cufft library be sufficient?</div>
<div><br>
</div>
<div>Or is it better to use the cufftw library supplied by NVIDIA? </div>
<div><br>
</div>
<div>Can I have some guidance on these aspects?</div>
<div><br>
</div>
<div>Thanks a lot and best regards,</div>
<div>Mauro Sgroi.</div>
<div><br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">___</font><font color="#000000" face="tahoma, sans-serif">____________________</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">Dr.
</span><b style="color:rgb(0,0,0)">Mauro Francesco Sgroi</b><br>
</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><font color="#000000"><font>Department of Chemistry
</font></font></font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">University of Turin</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">Via Quarello 15a</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000" face="tahoma, sans-serif">I-10135 TORINO (Italy)</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">Tel.</font></span></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-8372 </font></span></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><span style="color:rgb(0,0,0)"><font face="tahoma, sans-serif">+39 011-670-7364</font></span></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><span style="color:rgb(0,0,0)">e-mail:
</span><a href="mailto:maurofrancesco.sgroi@unito.it" target="_blank">maurofrancesco.sgroi@unito.it</a><br>
</font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font face="tahoma, sans-serif"><font color="#000000"><font>Web: </font></font></font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><a href="http://www.met.unito.it/" rel="noreferrer" style="color:rgb(17,85,204)" target="_blank">www.met.unito.it</a><br>
</p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><a href="http://www.chimica.unito.it" style="font-family:tahoma,sans-serif" target="_blank">www.chimica.unito.it</a><span style="color:rgb(0,0,0);font-family:tahoma,sans-serif"> </span><br>
</p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000"><font><font face="tahoma, sans-serif">Orcid:
<a href="https://orcid.org/0000-0002-0914-4217" target="_blank">https://orcid.org/0000-0002-0914-4217</a></font>
</font></font></p>
<p lang="zxx" style="margin-bottom:0cm;line-height:100%"><font color="#000000"><font>Webex: </font><a href="https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi" target="_blank">https://unito.webex.com/webappng/sites/unito/dashboard/pmr/maurofrancesco.sgroi</a> </font></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div></blockquote></div>