<div dir="ltr"><div>Are there fast communications between the two nodes? if not, the parallel distributed 3D FFT will be very slow (note the time taken by fft_scatt_yz). You might find convenient to exploit k-point parallelization, that requires much less communication: for instance, "mpirun -n 32 pw.x -nk 2" (2 pools of 16 processors, each pool performing parallel FFT), but you have to figure out a way to convince the first pool of 16 processors on node 1, the second on node 2 (or vice versa, as long as FFT parallelization happens inside a node, k-point parallelization across nodes )</div><div><br></div><div>Paolo<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Nov 5, 2020 at 7:29 PM Baer, Bradly via users <<a href="mailto:users@lists.quantum-espresso.org">users@lists.quantum-espresso.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Paolo,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thank you for your suggestion. I will add recompiling to move to 6.6 to my to do list. For now, I corrected the pseudopotential files as you indicated and the calculation ran successfully. It has become slightly faster, but still much slower than running
on a single node (3:30s vs 0:30s). Is there more that I should be doing to improve performance or is my test problem too small to see the benefits of parallelization?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Brad </div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-4475069457157334453Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
--------------------------------------------------------</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Bradly Baer</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">Graduate Research Assistant, Walker Lab</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">Interdisciplinary Materials Science</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Vanderbilt University</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
</div>
</div>
</div>
<div id="gmail-m_-4475069457157334453appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-4475069457157334453divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> users <<a href="mailto:users-bounces@lists.quantum-espresso.org" target="_blank">users-bounces@lists.quantum-espresso.org</a>> on behalf of Paolo Giannozzi <<a href="mailto:p.giannozzi@gmail.com" target="_blank">p.giannozzi@gmail.com</a>><br>
<b>Sent:</b> Thursday, November 5, 2020 10:01 AM<br>
<b>To:</b> Quantum ESPRESSO users Forum <<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
<b>Subject:</b> Re: [QE-users] Running efficiently on multiple nodes</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Thu, Nov 5, 2020 at 3:05 PM Baer, Bradly <<a href="mailto:bradly.b.baer@vanderbilt.edu" target="_blank">bradly.b.baer@vanderbilt.edu</a>> wrote:<br>
</div>
<div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div><i>Pseudo file Ga.pbe-dn-kjpaw_psl.1.0.0.UPF has been fixed on the fly.</i></div>
<div><i>To avoid this message in the future, permanently fix </i></div>
<div><i> your pseudo files following these instructions: </i></div>
<div><i><a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.com%2FQEF%2Fq-e%2Fblob%2Fmaster%2Fupftools%2Fhow_to_fix_upf.md&data=04%7C01%7Cbradly.b.baer%40vanderbilt.edu%7C73a5a5e938ac4ee11cb808d881a440f4%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C637401889751409012%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=511bORLQwH1OKUCoHg%2BbMb5I%2FiOEuHIFEbDlNl28678%3D&reserved=0" target="_blank">https://gitlab.com/QEF/q-e/blob/master/upftools/how_to_fix_upf.md</a></i></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>This is a possible source of trouble if the output directory is not visible to all processors. Please try one of the following:<br>
</div>
<div>- do what it is suggested (or simply: edit Ga.pbe-dn-kjpaw_psl.1.0.0.UPF, replace all occurrences of "&" with "&")</div>
<div>- get version 6.6, that reads the pseudopotential file on one processor and broadcast its contents to all other processes</div>
<div>- get the development version, that in addition is not sensitive to the presence of nonstandard "&" in the files,
</div>
<div><br>
</div>
<div>Paolo<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<br>
-Brad</div>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-4475069457157334453x_gmail-m_8263478242330334081Signature">
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
--------------------------------------------------------</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Bradly Baer</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">Graduate Research Assistant, Walker Lab</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif;background-color:rgb(255,255,255);display:inline">Interdisciplinary Materials Science</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Vanderbilt University</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
</div>
</div>
</div>
<div id="gmail-m_-4475069457157334453x_gmail-m_8263478242330334081appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-4475069457157334453x_gmail-m_8263478242330334081divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> users <<a href="mailto:users-bounces@lists.quantum-espresso.org" target="_blank">users-bounces@lists.quantum-espresso.org</a>>
on behalf of Paolo Giannozzi <<a href="mailto:p.giannozzi@gmail.com" target="_blank">p.giannozzi@gmail.com</a>><br>
<b>Sent:</b> Thursday, November 5, 2020 2:33 AM<br>
<b>To:</b> Quantum ESPRESSO users Forum <<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
<b>Subject:</b> Re: [QE-users] Running efficiently on multiple nodes</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">On Wed, Nov 4, 2020 at 11:28 PM Baer, Bradly <<a href="mailto:bradly.b.baer@vanderbilt.edu" target="_blank">bradly.b.baer@vanderbilt.edu</a>> wrote:<br>
</div>
<div>
<div> </div>
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Now that I have two nodes, the script for a single node results in a crash shortly after reading in the pseudopotentials.
</div>
</blockquote>
<div><br>
</div>
<div>which version of QE are you using, and which crash do you obtain, with which executable?<br>
</div>
<br clear="all">
</div>
Paolo<br>
<div>-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>
Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>
Phone +39-0432-558216, fax +39-0432-558222<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_________________<br>
Quantum ESPRESSO is supported by MaX (<a href="https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cbradly.b.baer%40vanderbilt.edu%7C73a5a5e938ac4ee11cb808d881a440f4%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C637401889751419008%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hFpaUoIPz4sX3DRUpP09MjX5XOrIVQEtwjR25n%2BUwlk%3D&reserved=0" rel="noreferrer" target="_blank">www.max-centre.eu</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
<a href="https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cbradly.b.baer%40vanderbilt.edu%7C73a5a5e938ac4ee11cb808d881a440f4%7Cba5a7f39e3be4ab3b45067fa80faecad%7C0%7C0%7C637401889751419008%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5BEAnLDjV8tgpwX7LRerSaKSiUsTwM8nRSoDNBcDJU4%3D&reserved=0" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a></blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>
Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>
Phone +39-0432-558216, fax +39-0432-558222<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu" rel="noreferrer" target="_blank">www.max-centre.eu</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a></blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>Phone +39-0432-558216, fax +39-0432-558222<br><br></div></div></div></div></div>