<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Louis,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I posted the input file for the system that runs slower on the GPU node versus CPU node:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<a href="https://github.com/jdh4/qe_container">https://github.com/jdh4/qe_container</a><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Can you tell by looking if that system should run slower on the GPU?</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
For using the local MPI libraries, I had the "Bind model" in mind:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<a href="https://sylabs.io/guides/3.7/user-guide/mpi.html">https://sylabs.io/guides/3.7/user-guide/mpi.html</a><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I guess that would put too much of a burden on the end users. Performance for the <span style="caret-color:rgb(0, 0, 0);background-color:rgb(255, 255, 255);display:inline !important">AUSURF112 benchmark is excellent with the container so no performance complaints
here.</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Jon</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> users <users-bounces@lists.quantum-espresso.org> on behalf of Louis Stuber via users <users@lists.quantum-espresso.org><br>
<b>Sent:</b> Friday, July 16, 2021 12:10 PM<br>
<b>To:</b> Quantum ESPRESSO users Forum <users@lists.quantum-espresso.org><br>
<b>Subject:</b> Re: [QE-users] [QE-GPU] Performance of the NGC Container</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:Wingdings}
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Verdana}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:blue;
text-decoration:underline}
p.x_MsoListParagraph, li.x_MsoListParagraph, div.x_MsoListParagraph
{margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle18
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
ol
{margin-bottom:0in}
ul
{margin-bottom:0in}
-->
</style>
<div lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal">Hi Jonathan,</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Thanks for your message and apologies for the late reply, as Paolo mentioned, the GPU version should never be slower than the CPU one except if it calls routines which are not implemented (fortunately the one you talked about has been
implemented recently).</p>
<p class="x_MsoNormal"> </p>
<ul type="disc" style="margin-top:0in">
<li class="x_MsoListParagraph" style="color:#201F1E; margin-left:0in"><span style="font-size:12.0pt; color:black">CUDA-aware MPI is nice. It appears that the container is configured to use the MPI libraries in the container instead of those installed for the
local cluster. Is this true? Can users take advantage of their local CUDA-aware MPI libraries?</span><span style="font-size:11.5pt"></span></li></ul>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Yes, a container will almost never see/use what’s on your local cluster except for low-level drivers/kernel things . It is not possible to use your own MPI installation without rebuilding the container, however, the container that was
uploaded on NGC already uses CUDA-aware MPI iirc so it should already perform well in that regard.</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Best,</p>
<p class="x_MsoNormal">Louis</p>
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b>From:</b> users <users-bounces@lists.quantum-espresso.org>
<b>On Behalf Of </b>Paolo Giannozzi<br>
<b>Sent:</b> Tuesday, July 6, 2021 8:47 PM<br>
<b>To:</b> Quantum ESPRESSO users Forum <users@lists.quantum-espresso.org><br>
<b>Subject:</b> Re: [QE-users] [QE-GPU] Performance of the NGC Container</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<table class="x_MsoNormalTable" border="1" cellspacing="5" cellpadding="0" style="background:#FFEB9C">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="x_MsoNormal"><b><span style="font-size:7.5pt; font-family:"Verdana",sans-serif; color:black">External email: Use caution opening links or attachments</span></b><span style="font-size:7.5pt; font-family:"Verdana",sans-serif; color:black">
</span></p>
</td>
</tr>
</tbody>
</table>
<p class="x_MsoNormal"> </p>
<div>
<div>
<div>
<p class="x_MsoNormal">The GPU acceleration of DFT-D3, using openacc, as well as its MPI parallelization, was implemented no more than a few days ago and will appear in the next release (soon). Apparently DFT-D3 takes a non-negligible amount of time. Without
MPI parallelization or GPU acceleration, it may easily become a bottleneck when running on many processors, or on GPUs.</p>
</div>
<div>
<p class="x_MsoNormal"> </p>
</div>
<div>
<p class="x_MsoNormal">Paolo</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<div>
<p class="x_MsoNormal">On Tue, Jul 6, 2021 at 7:44 PM Jonathan D. Halverson <<a href="mailto:halverson@princeton.edu">halverson@princeton.edu</a>> wrote:</p>
</div>
<blockquote style="border:none; border-left:solid #CCCCCC 1.0pt; padding:0in 0in 0in 6.0pt; margin-left:4.8pt; margin-right:0in">
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Hello (@Louis Stuber),</span></p>
</div>
<div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:11.5pt; color:#201F1E"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">The QE container on NGC (<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fngc.nvidia.com%2Fcatalog%2Fcontainers%2Fhpc%3Aquantum_espresso&data=04%7C01%7Clstuber%40nvidia.com%7C588fe1dda6464d020c5608d940ae9ada%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637611942291950905%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=hQAe0gUv1O2b8%2FCdTC29wwPXPH7Y2kaJCGoz1g7ioI4%3D&reserved=0" target="_blank">https://ngc.nvidia.com/catalog/containers/hpc:quantum_espresso</a>) appears
to be running very well for us on a node with two A100's for the "AUSURF112, Gold surface (112 atoms), DEISA pw" benchmark. We see a speed-up of 8x in comparison to running on 80 Skylake CPU-cores (no GPUs) where the code was built from source.</span><span style="font-size:11.5pt; color:#201F1E"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:11.5pt; color:#201F1E"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">The procedure we used for the above is here:</span><span style="font-size:11.5pt; color:#201F1E"></span></p>
<div>
<p class="x_MsoNormal"><span style="font-size:11.5pt; color:#201F1E"><a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresearchcomputing.princeton.edu%2Fsupport%2Fknowledge-base%2Fquantum-espresso&data=04%7C01%7Clstuber%40nvidia.com%7C588fe1dda6464d020c5608d940ae9ada%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637611942291960869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=daY1H57uDzVtMCk455MspB3VaIjabXlKnkTWrONGiEo%3D&reserved=0" target="_blank"><span style="font-size:12.0pt; color:black">https://researchcomputing.princeton.edu/support/knowledge-base/quantum-espresso</span></a></span></p>
</div>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:11.5pt; color:#201F1E"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">However, for one system we see a slow down (i.e., the code runs faster using only CPU-cores). Can you tell if the system below should perform well using the container?</span><span style="font-size:11.5pt; color:#201F1E"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:11.5pt; color:#201F1E"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">"My system is basically just two carbon dioxide molecules and doing a single point calculation on them using the PBE-D3 functional and basically just altering the distance between the two molecules
in the atomic coordinates."</span><span style="font-size:11.5pt; color:#201F1E"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:11.5pt; color:#201F1E"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Can someone comment in general on when one would expect the container running on GPUs to outperform a build-from-source executable running on CPU-cores?</span><span style="font-size:11.5pt; color:#201F1E"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:11.5pt; color:#201F1E"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">CUDA-aware MPI is nice. It appears that the container is configured to use the MPI libraries in the container instead of those installed for the local cluster. Is this true? Can users take advantage
of their local CUDA-aware MPI libraries?</span><span style="font-size:11.5pt; color:#201F1E"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:11.5pt; color:#201F1E"> </span></p>
</div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Jon</span></p>
</div>
</div>
</div>
<p class="x_MsoNormal">_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Clstuber%40nvidia.com%7C588fe1dda6464d020c5608d940ae9ada%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637611942291960869%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9R877U5bYrkH2UN%2FB6MtGzc7S9rzmbbuA8UMhEHGUk0%3D&reserved=0" target="_blank">www.max-centre.eu</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
<a href="https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Clstuber%40nvidia.com%7C588fe1dda6464d020c5608d940ae9ada%7C43083d15727340c1b7db39efd9ccc17a%7C0%7C0%7C637611942291970831%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=y4siAXAjRo1D%2BsTKHF9x%2B1J3RKaDZa%2FyN5%2BAdc0JYqo%3D&reserved=0" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a></p>
</blockquote>
</div>
<p class="x_MsoNormal"><br clear="all">
<br>
-- </p>
<div>
<div>
<div>
<div>
<div>
<p class="x_MsoNormal" style="margin-bottom:12.0pt">Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>
Univ. Udine, via delle Scienze 206, 33100 Udine, Italy<br>
Phone +39-0432-558216, fax +39-0432-558222</p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>