<div dir="ltr">Dear Ye, Dear Paolo,<br>I re-ran the benchmarks for my test case: a single MD step of a smallish supercell of a certain oxide semiconductor, with PBE and PAW (from PSlib). Previous timings were from the start of MD run until the end of the 1st SCF iteration of the 2nd MD step.<div><br></div><div>Interestingly, ELPA gave no advantage over ScaLAPACK, and diago_david_ndim=2 made things significantly slower. <div>The ScaLAPACK build is QE 6.5, the ELPA build is the development version from last month. Both compiled with Intel 2020 and Intel MPI.</div><div><br><div>Here are the numbers:<br><div></div></div></div><div><br></div><div><table border="0" cellpadding="0" cellspacing="0" width="610" style="border-collapse:collapse;width:457pt">
<colgroup><col width="99" style="width:74pt">
<col width="64" span="2" style="width:48pt">
<col width="115" style="width:86pt">
<col width="140" style="width:105pt">
<col width="64" span="2" style="width:48pt">
</colgroup><tbody><tr height="20" style="height:15pt">
<td height="20" width="99" style="height:15pt;width:74pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">MPI per node</td>
<td width="64" style="width:48pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">npool</td>
<td width="64" style="width:48pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">nodes</td>
<td width="115" style="width:86pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ELPA/Scalapack</td>
<td width="140" style="width:105pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">diago_david_ndim</td>
<td width="64" style="width:48pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">time / s</td>
<td width="64" style="width:48pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">speedup vs 1 node</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" align="right" style="height:15pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">56</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">1</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ELPA</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">1335</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap"></td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" align="right" style="height:15pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">56</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">1</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ELPA</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">2</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">1931</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap"></td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" align="right" style="height:15pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">56</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">1</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ScaLAPACK</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">976</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap"></td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" align="right" style="height:15pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">56</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">1</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ScaLAPACK</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">2</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">1486</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap"></td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" align="right" style="height:15pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">56</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ELPA</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">367</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">3.637602</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" align="right" style="height:15pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">56</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ELPA</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">2</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">729</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">2.648834</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" align="right" style="height:15pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">56</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ScaLAPACK</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">357</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">2.733894</td>
</tr>
<tr height="20" style="height:15pt">
<td height="20" align="right" style="height:15pt;padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">56</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">4</td>
<td style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">ScaLAPACK</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">2</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">555</td>
<td align="right" style="padding-top:1px;padding-right:1px;padding-left:1px;color:black;font-size:11pt;font-family:Calibri,sans-serif;vertical-align:bottom;border:none;white-space:nowrap">2.677477</td>
</tr>
</tbody></table><br></div><div>Best,</div></div><div>Michal</div></div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 27 May 2020 at 15:47, Ye Luo <<a href="mailto:xw111luoye@gmail.com">xw111luoye@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>3.26x seems possible to me. It can be caused by load imbalance in the iterative solver among the 4 k-points.</div><div></div><div>Could you list the time in seconds with 1 node and 4 nodes? Those you used to calculate 3.26x.</div><div>Could you also try diago_david_ndim=2 under "&ELECTRONS" and provide 1 and 4-node time in seconds?</div><div><br></div><div><div></div><div>In addition, you may try ELPA which usually gives better performance than scalapack.</div><div><br></div></div><div>Thanks,<br></div><div>Ye</div><div><div><div dir="ltr"><div dir="ltr"><div><div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>Computational Science Division & Leadership Computing Facility<br>
Argonne National Laboratory</div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 27, 2020 at 9:27 AM Michal Krompiec <<a href="mailto:michal.krompiec@gmail.com" target="_blank">michal.krompiec@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello,<br>
How can I minimize inter-node MPI communication in a pw.x run? My<br>
system doesn't have Infiniband and inter-node MPI can easily become<br>
the bottleneck.<br>
Let's say, I'm running a calculation with 4 k-points, on 4 nodes, with<br>
56 MPI tasks per node. I would then use -npool 4 to create 4 pools for<br>
the k-point parallelization. However, it seems that the<br>
diagonalization is by default parallelized imperfectly (or isn't it?):<br>
Subspace diagonalization in iterative solution of the eigenvalue problem:<br>
one sub-group per band group will be used<br>
scalapack distributed-memory algorithm (size of sub-group: 7* 7 procs)<br>
So far, speedup on 4 nodes vs 1 node is 3.26x. Is it normal or does it<br>
look like it can be improved?<br>
<br>
Best regards,<br>
<br>
Michal Krompiec<br>
Merck KGaA<br>
Southampton, UK<br>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu/quantum-espresso" rel="noreferrer" target="_blank">www.max-centre.eu/quantum-espresso</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
</blockquote></div>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu/quantum-espresso" rel="noreferrer" target="_blank">www.max-centre.eu/quantum-espresso</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a></blockquote></div>