<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-family:georgia,serif;font-size:small">Hello Pietro,</div><div class="gmail_default" style="font-family:georgia,serif;font-size:small"><br></div><div class="gmail_default" style="font-family:georgia,serif;font-size:small">This is very helpful. Thank you for the prompt response. I have consulted with the maintainers of Spack QE as well, and they have noted that the nvhpc module is required (as expected), despite not being listed as a dependency. Conversely, it seems that the cuda module that <i>is</i> listed is scarcely applicable. I will be proceeding with this route.<br><br>I appreciate your help!<br><br>Best,</div><div class="gmail_default" style="font-family:georgia,serif;font-size:small">Hiro</div></div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><b><font face="georgia, serif">Hironori Kondo</font></b><div><font face="georgia, serif">Harvard College | Class of 2025</font></div><div><font face="georgia, serif">A.B. Candidate in Applied Mathematics</font></div><div><font face="georgia, serif">Concurrent A.M. in Chemistry</font></div><div><a href="mailto:hirokondo@college.harvard.edu" style="font-family:georgia,serif;font-size:x-small" target="_blank">hirokondo@college.harvard.edu</a> | <a href="mailto:hkondo@mit.edu" style="font-family:georgia,serif;font-size:x-small" target="_blank">hkondo@mit.edu</a></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 10, 2024 at 3:56 AM Pietro Davide Delugas <<a href="mailto:pdelugas@sissa.it" target="_blank">pdelugas@sissa.it</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
Dear Hironori</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
If you have Nvidia GPUs, the efficient and reliable way to use QE on them is to compile the nvhpc package. </div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
In that toolchain, most of the suite is accelerated and well-tested.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
About the toolchain for compiling QE is rather short</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
you need an nvhpc suite (versions 23.4 or 23.11 are recommended)</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
any FFTW3 library with nvfortran support</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
any BLAS/LAPACK library with nvfortran support</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
e.g. you could use directly MKL with nvfortran support that includes both, but again any working version of those libraries would be ok</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
OpenMPI library usually comes with the NVHPC package.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
with these components, you can compile QE either using GNU autoconf (see <a href="https://gitlab.com/QEF/q-e/-/wikis/Developers/Make-build-system" id="m_1238252801644646192m_2673870782107090599LPlnk719675" title="https://gitlab.com/QEF/q-e/-/wikis/Developers/Make-build-system" target="_blank">
here)</a> or CMake (see <a href="https://gitlab.com/QEF/q-e/-/wikis/Developers/CMake-build-system" id="m_1238252801644646192m_2673870782107090599LPlnk321496" title="https://gitlab.com/QEF/q-e/-/wikis/Developers/CMake-build-system" target="_blank">
here</a> )</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
Indeed using spack is a straightforward way to include and verify the whole toolchain directly and then compile with CMake build system; it is probably better to have nvhpc already installed in the system and configure spack for using it.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
ASE uses QE as an IO calculator, so the two toolchains should be independent, no need to integrate them in conda or any kind of environment.</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
Hope it helps</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
best regards</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
Pietro</div>
<div id="m_1238252801644646192m_2673870782107090599appendonsend"></div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif;font-size:11pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_1238252801644646192m_2673870782107090599divRplyFwdMsg" dir="ltr"><span style="font-family:Calibri,sans-serif;font-size:11pt;color:rgb(0,0,0)"><b>From:</b> developers <<a href="mailto:developers-bounces@lists.quantum-espresso.org" target="_blank">developers-bounces@lists.quantum-espresso.org</a>> on behalf of Hironori Kondo <<a href="mailto:hirokondo@college.harvard.edu" target="_blank">hirokondo@college.harvard.edu</a>><br>
<b>Sent:</b> Wednesday, October 9, 2024 17:01<br>
<b>To:</b> <a href="mailto:developers@lists.quantum-espresso.org" target="_blank">developers@lists.quantum-espresso.org</a> <<a href="mailto:developers@lists.quantum-espresso.org" target="_blank">developers@lists.quantum-espresso.org</a>><br>
<b>Subject:</b> [QE-developers] Best Practices for GPU HPC Implementation (Spack, NCG Singularity, etc.)</span>
<div> </div>
</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt">Hello QE dev team,</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt"><br>
</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt">I hope this email finds you well. I am working on implementing QE in an HPC environment, with GPU functionality. I know there are many questions of this variety in the archives, but
I think my question is more specific and has not been discussed thus far. My university HPC support team recommended that I contact the QE primary authors for the best guidance.</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt"><br>
</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt">From what I've seen, most people are installing QE with CUDA using the standard GNU toolchain-style process with HPC SDK. My university is not too keen on this, and recommends something
tidier, if possible---Singularity, Mamba/Conda, Spack, etc.<br>
<br>
On the Singularity front, the latest image offered by Nvidia NCG is 7.1. I suspect this is due to the transition to OpenAcc, but that's just a guess. I would like to stick with 7.3.1, especially with improvements in EPW. </div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt"><br>
</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt">On the Conda front, there doesn't seem to be a GPU option by default. I could try building within my Mamba environment with the standard process, but that seems less neat. </div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt"><br>
</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt">On the Spack front, I see that there is QE 7.3.1, and there is a CUDA variant:
<a href="https://packages.spack.io/package.html?name=quantum-espresso#" id="m_1238252801644646192m_2673870782107090599OWA0d006844-e931-bb43-a34c-21c428674cc5" target="_blank">
https://packages.spack.io/package.html?name=quantum-espresso#</a><br>
However, it relies on the CUDA package (linked within the Spack page above), not the separate HPC SDK package:
<a href="https://packages.spack.io/package.html?name=nvhpc" id="m_1238252801644646192m_2673870782107090599OWA181085c7-1d8c-aaa0-ec1f-8ce0b4fce3bf" target="_blank">
https://packages.spack.io/package.html?name=nvhpc</a><br>
I'm not sure what implications this has. </div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt"><br>
</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt"><br>
</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt">I would be much obliged for your guidance on the best approach here. If it is relevant, I am managing my HPC workflow in quacc by Princeton's Andrew Rosen, which interfaces with QE
via ASE.</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt"><br>
</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt">Best,</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt">Hiro</div>
<div style="direction:ltr;font-family:georgia,serif;font-size:9.75pt"><br>
</div>
<div style="direction:ltr;font-family:georgia,serif"><b>Hironori Kondo</b></div>
<div style="direction:ltr;font-family:georgia,serif">Harvard College | Class of 2025</div>
<div style="direction:ltr;font-family:georgia,serif">A.B. Candidate in Applied Mathematics</div>
<div style="direction:ltr;font-family:georgia,serif">Concurrent A.M. in Chemistry</div>
<div style="direction:ltr"><span style="font-family:georgia,serif;font-size:7.5pt"><a href="mailto:hirokondo@college.harvard.edu" id="m_1238252801644646192m_2673870782107090599OWA72d558b8-02c9-568c-5a5a-3ef4cf913656" target="_blank">hirokondo@college.harvard.edu</a></span> |
<span style="font-family:georgia,serif;font-size:7.5pt"><a href="mailto:hkondo@mit.edu" id="m_1238252801644646192m_2673870782107090599OWA173f9779-852f-f59a-62f2-0391680c13c3" target="_blank">hkondo@mit.edu</a></span></div>
</div>
</div></blockquote></div>