<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><blockquote type="cite"><div><blockquote type="cite">I found when I took the 432 atom system I sent you, and ran it on 128 cores in smp mode<br></blockquote><blockquote type="cite">(1 MPI process/node - 2 GB per process) it did work (-ntg 32 -ndiag 121<br></blockquote><br>32 task groups? that's a lot</div></blockquote><div><br></div><div>Yeah - I ran that by mistake. But I figured I'd mention it anyway</div><div><br></div><br><blockquote type="cite"><div><br><br><blockquote type="cite">as well as -ntg 4 -ndiag 121)<br></blockquote><br>4 looks more reasonable in my opinion<br><br></div></blockquote><div><br></div><div>Right.</div><div><br></div><br><blockquote type="cite"><div><blockquote type="cite">- the system didn't fit into memory in vn mode (4 mpi processes/node -<br></blockquote><blockquote type="cite">512 MB per process)<br></blockquote><br>that job requires approx. 100Mb of dynamically allocated RAM per process, plus<br>a few tens of Mb of work space. Why it does not fit into 512Mb is a mystery,<br>unless each process comes with a copy of all libraries. If this is the case, the<br>maximum you can fit into 512Mb is a code printing "Hello world" in parallel.<br><br></div></blockquote><div><br></div><div>Yeah - I am very curious about this. I was actually *very* surprised when this happened.</div><div><br></div><br><blockquote type="cite"><div>By the way: the default number of bands in metallic calculations can be trimmed<br>by a significant amount (e.g. 500 instead of 576)<br><br><blockquote type="cite">I then tried the system in dual mode (2 mpi processes/node - 1 GB per process)<br></blockquote><blockquote type="cite">using -ntg 4 and -ndiag 121. In this case, the cholesky error came up:<br></blockquote><br><br>the code performs exactly the same operations, independently on how the MPI<br>processes are distributed. It looks like yet another BlueGene weirdness, like this:<br> <a href="http://www.democritos.it:8888/O-sesame/chngview?cn=5777">http://www.democritos.it:8888/O-sesame/chngview?cn=5777</a><br> <a href="http://www.democritos.it:8888/O-sesame/chngview?cn=5932">http://www.democritos.it:8888/O-sesame/chngview?cn=5932</a><br>that however affected only the internal parallel diagonalization, not the new<br>scalapack algorithm. I do not see any evidence that there is anything wrong<br>with the code itself.<br><br></div></blockquote><div><br></div><div>OK - so perhaps I should try this same run with scalapack enabled instead just to be sure of what happens and that I haven't miscompiled anything. That is only available in the latest CVS versions right? </div><div><br></div><div>Dave</div><div><br></div><div><br></div></div><div apple-content-edited="true"> <span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 14px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; font-size: 12px; ">David E. Farrell</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px; ">Post-Doctoral Fellow</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font class="Apple-style-span" size="3"><span class="Apple-style-span" style="font-size: 12px; ">Department of Materials Science and Engineering</span></font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; font-size: 12px; ">Northwestern University</font></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px; "><font face="Helvetica" size="3" style="font: normal normal normal 12px/normal Helvetica; font-size: 12px; ">email: <a href="mailto:d-farrell2@northwestern.edu">d-farrell2@northwestern.edu</a></font></div></div></div></span> </div><br></body></html>