<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><br class=""></div>Works with 5.2.0 (Cray PrgEnv-gnu, fortran 4.9, Cray libsci) with -ndiag > 1, -ntg > 1. Thanks! Lesson learned…<div class=""><br class=""></div><div class="">Kane</div><div class=""><br class=""></div><div class=""><br class=""><div class=""><div><blockquote type="cite" class=""><div class="">On 21 Oct 2015, at 17:27, Paolo Giannozzi <<a href="mailto:p.giannozzi@gmail.com" class="">p.giannozzi@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><div class="">Unless you need new developments that are available in the svn version only, please try if it works with the 5.2.0 version. We just found a problem (also affecting v.5.2.1) with "task groups" that may lead to strange crashes.<br class=""><br class=""></div>Paolo<br class=""></div><div class="gmail_extra" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class=""><div class="gmail_quote">On Wed, Oct 21, 2015 at 11:04 AM, Kane O'Donnell<span class="Apple-converted-space"> </span><span dir="ltr" class=""><<a href="mailto:kane.odonnell@gmail.com" target="_blank" class="">kane.odonnell@gmail.com</a>></span><span class="Apple-converted-space"> </span>wrote:<br class=""><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><div style="word-wrap: break-word;" class=""><div class=""><br class=""></div>Hi all,<br class=""><div class=""><div style="word-wrap: break-word;" class=""><div class=""><br class=""></div><div class="">Wondering if I can get some help trying to diagnose a crash. I’m running the SVN latest on a Cray XC40 (Magnus - <a href="https://www.pawsey.org.au/our-systems/magnus-technical-specifications/" target="_blank" class="">https://www.pawsey.org.au/our-systems/magnus-technical-specifications/</a>). Usually no problems, but I have difficulties getting the attached slab calculation to run past the first few davidson diagonalizations. It’s a 3x3 c-oriented slab of Na3Bi, the minimum I can use to capture a certain adsorbate reconstruction (this just the bare slab). It’s only a 72 atom cell but Bi has a lot of electrons (I think there’s about ~800 electrons and ~900 bands). I have spin-orbit coupling switched on (important for this solid), and I have been able to do calculations on the smaller unit cell using the library pseudopotentials listed in the species block. Calculations on systems of this size (e.g. O(1000) electrons, bands) are routine on Magnus, so I think I’m probably just doing something stupid but can’t seem to figure it out.</div><div class=""><br class=""></div><div class="">Typical run conditions are with 384 processors (16 nodes, 24 cores), with -nk 3 -ndiag 100 -ntg 8. Moving down to ~12 nodes leads to a out of memory crash just as the code reports it is allocating random wf’s at the beginning. From 16 nodes and upwards, the crashes happen around the diagonalization step. Switching to CG works but the slowdown is astronomical (~10000 seconds per SCF step, not feasible for a relaxation). A typical output is attached. With -ndiag > 1, the error is “problems computing cholesky”, with -ndiag = 1, the error is "S matrix not positive definite”, both from cdiaghg. A search of the forums suggests this issue comes up every now and then on wildly different systems and is usually blamed on the user/compiler/lapack/blas/scalapack. So, details: QE was compiled by me on Magnus with PrgEnv-gnu (fortran 4.9.0) against the Cray libsci (includes fftw, scalapack, etc), with:</div><div class=""><br class=""></div><div class="">./configure —enable-parallel —with-scalapack=yes FC=ftn CC=cc </div><div class=""><br class=""></div><div class="">and all tests are passed with no problems.</div><div class=""><br class=""></div><div class="">Any ideas? Let me know if there is any further information necessary.</div><div class=""><br class=""></div><div class="">Best regards,</div><div class=""><br class=""></div><div class="">Kane<br class=""><div class=""><div style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><p class="MsoNormal"><span lang="EN-US" class=""><font face="Arial" class=""><b class="">Kane O'Donnell</b></font></span><font face="Arial" class=""><span lang="EN-US" style="font-size: 8pt;" class=""><br class=""></span><span lang="EN-US" style="color: rgb(172, 132, 4);" class=""><span style="font-size: 9px;" class=""><b class="">Postdoctoral Research Fellow | Department of Physics, Astronomy and Medical Radiation Science</b></span><br class=""></span><span lang="EN-US" class=""><br class=""></span><span style="font-size: 9px;" class=""><span lang="EN-US" class=""><b class="">Curtin University</b><br class=""><span style="color: rgb(172, 132, 4);" class=""><b class="">Tel |</b></span> </span><span lang="EN-US" class=""><a href="tel:%2B61%208%209266%201381" value="+61892661381" target="_blank" class="">+61 8 9266 1381</a></span><span lang="EN-US" class=""> </span><span lang="EN-US" class=""><br class=""></span><span lang="EN-US" style="color: rgb(172, 132, 4);" class=""><b class="">Fax |</b></span><span lang="EN-US" class=""> </span><span lang="EN-US" class=""><a href="tel:%2B61%208%209266%202377" value="+61892662377" target="_blank" class="">+61 8 9266 2377</a></span><span lang="EN-US" class=""> <br class=""><br class=""></span><span lang="EN-US" style="color: rgb(172, 132, 4);" class=""><b class="">Email |</b></span><span lang="EN-US" class=""> </span><span lang="EN-US" class=""><a class="">kane.odonnell@curtin.edu.au</a></span></span></font></p><br class=""></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><span class=""></span></span></span></div></div></div></div></div><br class=""><div style="word-wrap: break-word;" class=""><div class=""><div style="word-wrap: break-word;" class=""><div class=""><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><span class=""></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;" class=""><span style="font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; font-family: Arial; font-size: 9px;" class="">Curtin University is a trademark of Curtin University of Technology</span><br style="font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; font-family: Arial; font-size: 9px;" class=""><span style="font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; font-family: Arial; font-size: 9px;" class="">CRICOS Provider Code 00301J</span></span></div><br class=""></div><div class=""><div class="">&control</div><div class=""> <span class="Apple-converted-space"> </span>calculation = 'relax',</div><div class=""> <span class="Apple-converted-space"> </span>title = '',</div><div class=""> <span class="Apple-converted-space"> </span>outdir = './',</div><div class=""> <span class="Apple-converted-space"> </span>prefix = 'Na3Bi_331',</div><div class=""> <span class="Apple-converted-space"> </span>pseudo_dir = '/group/partner1197/kodonnell/qe_pseudos/PSEUDOPOTENTIALS/',</div><div class=""> <span class="Apple-converted-space"> </span>wf_collect = .true.</div><div class="">/</div><div class="">&system</div><div class=""> <span class="Apple-converted-space"> </span>ibrav = 0,</div><div class=""> <span class="Apple-converted-space"> </span>nat = 72,</div><div class=""> <span class="Apple-converted-space"> </span>ntyp = 2,</div><div class=""> <span class="Apple-converted-space"> </span>nbnd = 896,</div><div class=""> <span class="Apple-converted-space"> </span>ecutwfc = 50,</div><div class=""> <span class="Apple-converted-space"> </span>!ecutrho = 280,</div><div class=""> <span class="Apple-converted-space"> </span>!tot_charge=+1.0,</div><div class=""> <span class="Apple-converted-space"> </span>occupations = 'smearing',</div><div class=""> <span class="Apple-converted-space"> </span>smearing = 'mv',</div><div class=""> <span class="Apple-converted-space"> </span>degauss = 0.0073,</div><div class=""> <span class="Apple-converted-space"> </span>lspinorb = .true.,</div><div class=""> <span class="Apple-converted-space"> </span>noncolin = .true.,</div><div class=""> <span class="Apple-converted-space"> </span>starting_magnetization(1) = 0.0,</div><div class=""> <span class="Apple-converted-space"> </span>starting_magnetization(2) = 0.0</div><div class="">/</div><div class="">&electrons</div><div class=""> <span class="Apple-converted-space"> </span>conv_thr = 1.0D-7</div><div class="">/</div><div class="">&ions</div><div class="">/</div><div class="">ATOMIC_SPECIES</div><div class=""> <span class="Apple-converted-space"> </span>Bi 1.0 Bi.rel-pbe-dn-kjpaw_psl.0.2.2.UPF</div><div class=""> <span class="Apple-converted-space"> </span>Na 1.0 Na.rel-pbe-spn-kjpaw_psl.0.2.UPF</div><div class="">CELL_PARAMETERS angstrom</div><div class=""> <span class="Apple-converted-space"> </span>16.344 0 0</div><div class=""> <span class="Apple-converted-space"> </span>-8.172 14.1543 0</div><div class=""> <span class="Apple-converted-space"> </span>1.77359e-15 3.07196e-15 28.965</div><div class="">K_POINTS automatic</div><div class=""> <span class="Apple-converted-space"> </span>2 2 1 0 0 0</div><div class="">ATOMIC_POSITIONS angstrom</div><div class="">Bi 0 3.146 2.414 0 0 0</div><div class="">Na 2.724 4.718 2.414 0 0 0</div><div class="">Na 0 3.146 5.629</div><div class="">Bi 2.724 1.573 7.241</div><div class="">Na 2.724 4.718 7.241</div><div class="">Na 2.724 1.573 0.801 0 0 0</div><div class="">Na 2.724 1.573 4.026</div><div class="">Na 0 3.146 8.854</div><div class="">Bi -2.724 7.864 2.414 0 0 0</div><div class="">Na 0 9.436 2.414 0 0 0</div><div class="">Na -2.724 7.864 5.629</div><div class="">Bi 0 6.291 7.241</div><div class="">Na 0 9.436 7.241</div><div class="">Na 0 6.291 0.801 0 0 0</div><div class="">Na 0 6.291 4.026</div><div class="">Na -2.724 7.864 8.854</div><div class="">Bi -5.448 12.582 2.414 0 0 0</div><div class="">Na -2.724 14.154 2.414 0 0 0</div><div class="">Na -5.448 12.582 5.629</div><div class="">Bi -2.724 11.009 7.241</div><div class="">Na -2.724 14.154 7.241</div><div class="">Na -2.724 11.009 0.801 0 0 0</div><div class="">Na -2.724 11.009 4.026</div><div class="">Na -5.448 12.582 8.854</div><div class="">Bi 5.448 3.146 2.414 0 0 0</div><div class="">Na 8.172 4.718 2.414 0 0 0</div><div class="">Na 5.448 3.146 5.629</div><div class="">Bi 8.172 1.573 7.241</div><div class="">Na 8.172 4.718 7.241</div><div class="">Na 8.172 1.573 0.801 0 0 0</div><div class="">Na 8.172 1.573 4.026</div><div class="">Na 5.448 3.146 8.854</div><div class="">Bi 2.724 7.864 2.414 0 0 0</div><div class="">Na 5.448 9.436 2.414 0 0 0</div><div class="">Na 2.724 7.864 5.629</div><div class="">Bi 5.448 6.291 7.241</div><div class="">Na 5.448 9.436 7.241</div><div class="">Na 5.448 6.291 0.801 0 0 0</div><div class="">Na 5.448 6.291 4.026</div><div class="">Na 2.724 7.864 8.854</div><div class="">Bi 0 12.582 2.414 0 0 0</div><div class="">Na 2.724 14.154 2.414 0 0 0</div><div class="">Na 0 12.582 5.629</div><div class="">Bi 2.724 11.009 7.241</div><div class="">Na 2.724 14.154 7.241</div><div class="">Na 2.724 11.009 0.801 0 0 0</div><div class="">Na 2.724 11.009 4.026</div><div class="">Na 0 12.582 8.854</div><div class="">Bi 10.896 3.146 2.414 0 0 0</div><div class="">Na 13.62 4.718 2.414 0 0 0</div><div class="">Na 10.896 3.146 5.629</div><div class="">Bi 13.62 1.573 7.241</div><div class="">Na 13.62 4.718 7.241</div><div class="">Na 13.62 1.573 0.801 0 0 0</div><div class="">Na 13.62 1.573 4.026</div><div class="">Na 10.896 3.146 8.854</div><div class="">Bi 8.172 7.864 2.414 0 0 0</div><div class="">Na 10.896 9.436 2.414 0 0 0</div><div class="">Na 8.172 7.864 5.629</div><div class="">Bi 10.896 6.291 7.241</div><div class="">Na 10.896 9.436 7.241</div><div class="">Na 10.896 6.291 0.801 0 0 0</div><div class="">Na 10.896 6.291 4.026</div><div class="">Na 8.172 7.864 8.854</div><div class="">Bi 5.448 12.582 2.414 0 0 0</div><div class="">Na 8.172 14.154 2.414 0 0 0</div><div class="">Na 5.448 12.582 5.629</div><div class="">Bi 8.172 11.009 7.241</div><div class="">Na 8.172 14.154 7.241</div><div class="">Na 8.172 11.009 0.801 0 0 0</div><div class="">Na 8.172 11.009 4.026</div><div class="">Na 5.448 12.582 8.854</div></div><div class=""></div></div></div></div><br class=""><div style="word-wrap: break-word;" class=""><div class=""><div style="word-wrap: break-word;" class=""><div class=""></div><div class=""><br class=""></div></div></div><br class=""></div><br class="">_______________________________________________<br class="">Pw_forum mailing list<br class=""><a href="mailto:Pw_forum@pwscf.org" class="">Pw_forum@pwscf.org</a><br class=""><a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank" class="">http://pwscf.org/mailman/listinfo/pw_forum</a><br class=""></blockquote></div><br class=""><br clear="all" class=""><br class="">--<span class="Apple-converted-space"> </span><br class=""><div class="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><span class=""><span class=""><font color="#888888" class="">Paolo Giannozzi, Dept. Chemistry&Physics&Environment,<br class="">Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br class="">Phone<span class="Apple-converted-space"> </span><a href="tel:%2B39-0432-558216" value="+390432558216" target="_blank" class="">+39-0432-558216</a>, fax<span class="Apple-converted-space"> </span><a href="tel:%2B39-0432-558222" value="+390432558222" target="_blank" class="">+39-0432-558222</a></font></span></span></div></div></div></div></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Pw_forum mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="mailto:Pw_forum@pwscf.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Pw_forum@pwscf.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://pwscf.org/mailman/listinfo/pw_forum" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">http://pwscf.org/mailman/listinfo/pw_forum</a></div></blockquote></div><br class=""></div></div></body></html>