<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Dear Hironori</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
If you have Nvidia GPUs, the efficient and reliable way to use QE on them is to compile the nvhpc package. </div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
In that toolchain, most of the suite is accelerated and well-tested.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
About the toolchain for compiling QE is rather short</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
you need an nvhpc suite (versions 23.4 or 23.11 are recommended)</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
any FFTW3 library with nvfortran support</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
any BLAS/LAPACK library with nvfortran support</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
e.g. you could use directly MKL with nvfortran support that includes both, but again any working version of those libraries would be ok</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
OpenMPI library usually comes with the NVHPC package.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
with these components, you can compile QE either using GNU autoconf (see <a href="https://gitlab.com/QEF/q-e/-/wikis/Developers/Make-build-system" id="LPlnk719675" title="https://gitlab.com/QEF/q-e/-/wikis/Developers/Make-build-system">
here)</a> or CMake (see <a href="https://gitlab.com/QEF/q-e/-/wikis/Developers/CMake-build-system" id="LPlnk321496" title="https://gitlab.com/QEF/q-e/-/wikis/Developers/CMake-build-system">
here</a> )</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Indeed using spack is a straightforward way to include and verify the whole toolchain directly and then compile with CMake build system; it is probably better to have nvhpc already installed in the system and configure spack for using it.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
ASE uses QE as an IO calculator, so the two toolchains should be independent, no need to integrate them in conda or any kind of environment.</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Hope it helps</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
best regards</div>
<div class="elementToProof" style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
Pietro</div>
<div id="appendonsend"></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">
<br>
</div>
<hr style="display: inline-block; width: 98%;">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> developers <developers-bounces@lists.quantum-espresso.org> on behalf of Hironori Kondo <hirokondo@college.harvard.edu><br>
<b>Sent:</b> Wednesday, October 9, 2024 17:01<br>
<b>To:</b> developers@lists.quantum-espresso.org <developers@lists.quantum-espresso.org><br>
<b>Subject:</b> [QE-developers] Best Practices for GPU HPC Implementation (Spack, NCG Singularity, etc.)</span>
<div> </div>
</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;">Hello QE dev team,</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;"><br>
</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;">I hope this email finds you well. I am working on implementing QE in an HPC environment, with GPU functionality. I know there are many questions of this variety in the archives, but
I think my question is more specific and has not been discussed thus far. My university HPC support team recommended that I contact the QE primary authors for the best guidance.</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;"><br>
</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;">From what I've seen, most people are installing QE with CUDA using the standard GNU toolchain-style process with HPC SDK. My university is not too keen on this, and recommends something
tidier, if possible---Singularity, Mamba/Conda, Spack, etc.<br>
<br>
On the Singularity front, the latest image offered by Nvidia NCG is 7.1. I suspect this is due to the transition to OpenAcc, but that's just a guess. I would like to stick with 7.3.1, especially with improvements in EPW. </div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;"><br>
</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;">On the Conda front, there doesn't seem to be a GPU option by default. I could try building within my Mamba environment with the standard process, but that seems less neat. </div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;"><br>
</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;">On the Spack front, I see that there is QE 7.3.1, and there is a CUDA variant:
<a href="https://packages.spack.io/package.html?name=quantum-espresso#" id="OWA0d006844-e931-bb43-a34c-21c428674cc5" class="OWAAutoLink" data-auth="NotApplicable">
https://packages.spack.io/package.html?name=quantum-espresso#</a><br>
However, it relies on the CUDA package (linked within the Spack page above), not the separate HPC SDK package:
<a href="https://packages.spack.io/package.html?name=nvhpc" id="OWA181085c7-1d8c-aaa0-ec1f-8ce0b4fce3bf" class="OWAAutoLink" data-auth="NotApplicable">
https://packages.spack.io/package.html?name=nvhpc</a><br>
I'm not sure what implications this has. </div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;"><br>
</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;"><br>
</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;">I would be much obliged for your guidance on the best approach here. If it is relevant, I am managing my HPC workflow in quacc by Princeton's Andrew Rosen, which interfaces with QE
via ASE.</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;"><br>
</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;">Best,</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;">Hiro</div>
<div style="direction: ltr; font-family: georgia, serif; font-size: 9.75pt;"><br>
</div>
<div style="direction: ltr; font-family: georgia, serif;"><b>Hironori Kondo</b></div>
<div style="direction: ltr; font-family: georgia, serif;">Harvard College | Class of 2025</div>
<div style="direction: ltr; font-family: georgia, serif;">A.B. Candidate in Applied Mathematics</div>
<div style="direction: ltr; font-family: georgia, serif;">Concurrent A.M. in Chemistry</div>
<div style="direction: ltr;"><span style="font-family: georgia, serif; font-size: 7.5pt;"><a href="mailto:hirokondo@college.harvard.edu" id="OWA72d558b8-02c9-568c-5a5a-3ef4cf913656" class="OWAAutoLink">hirokondo@college.harvard.edu</a></span> |
<span style="font-family: georgia, serif; font-size: 7.5pt;"><a href="mailto:hkondo@mit.edu" id="OWA173f9779-852f-f59a-62f2-0391680c13c3" class="OWAAutoLink">hkondo@mit.edu</a></span></div>
</body>
</html>