<div dir="ltr">I forgot to forward my replies to W90 mailing lists.<br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <strong class="gmail_sendername" dir="auto">H. Lee</strong> <span dir="auto"><<a href="mailto:hjunlee@gmail.com">hjunlee@gmail.com</a>></span><br>Date: Mon, Mar 1, 2021 at 10:40 AM<br>Subject: Re: [Wannier] MPI version large systems<br>To: Jonathan Backman <<a href="mailto:jbackman@iis.ee.ethz.ch">jbackman@iis.ee.ethz.ch</a>><br></div><br><br><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Dear Jonathan:</div><div dir="ltr"><br></div><div>I think that you have an issue with reading W90 inputs when using 27 cores even if you have enough memory.</div><div><br></div><div>I regenerated all relevant inputs (*.mmn, *.amn, and *.eig) to W90 using your *.wout and reproduced your issue.</div><div>Of course, I just regenerated inputs with the same file size, but contained random numbers; however, they are surely enough for the test.</div><div><br></div><div>I performed the test on the one node with 2 Intel Broadwell processors running at 2.6 GHz, with 14 cores each (28 cores per node), 128 GB memory, and GPFS file system.</div><div>In this case, I succeeded in reading inputs using 10 cores as you can see below:</div><div><br></div><div>(START OF OUTPUT)</div><div><span style="color:rgb(0,0,0);font-family:Menlo;font-size:11px"> +----------------------------------------------------------------------------+</span><br></div><div>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | b_k Vectors (Ang^-1) and Weights (Ang^2) |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | ---------------------------------------- |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | No. b_k(x) b_k(y) b_k(z) w_b |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | --- -------------------------------- -------- |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 1 0.095739 0.000000 0.000000 54.549228 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 2 0.000000 0.095739 0.000000 54.549228 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 3 0.000000 0.000000 0.095739 54.549228 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 4 0.000000 0.000000 -0.095739 54.549228 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 5 0.000000 -0.095739 0.000000 54.549228 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 6 -0.095739 0.000000 0.000000 54.549228 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> +----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | b_k Directions (Ang^-1) |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | ----------------------- |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | No. x y z |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | --- -------------------------------- |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 1 0.095739 0.000000 0.000000 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 2 0.000000 0.095739 0.000000 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | 3 0.000000 0.000000 0.095739 |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> +----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> Time to get kmesh 0.022 (sec)</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> *============================================================================*</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | MEMORY ESTIMATE |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | Maximum RAM allocated during each phase of the calculation |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> *============================================================================*</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | Disentanglement 34494.77 Mb |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | Wannierise: 30250.47 Mb |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> | plot_wannier: 30250.47 Mb |</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> *----------------------------------------------------------------------------*</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> Starting a new Wannier90 calculation ...</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> Reading overlaps from wannier90.mmn : Created on</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> Reading projections from wannier90.amn : Created on</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> Time to read overlaps 1009.729 (sec)</span></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br></p>
<p style="margin:0px;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"> *------------------------------- DISENTANGLE --------------------------------*</span></p></div><div>(END OF OUTPUT)</div><div><br></div><div>But, when I use the cores larger than 12, I encountered the same issue and confirmed that the memory footprint increases as the number of cores used increases.</div><div>At this time, I am not sure whether this is due to the increase of internal MPI memory usage or some kind of buffer for the file system; this is specific to the system configuration.</div><div><br></div><div>The only thing I can tell you is that with your inputs with very large file size the memory footprint increases with the number of cores used.</div><div>So I would suggest you to try to reduce the number of cores to the smaller one, for example, 8 or 12, etc.</div><div><br></div><div>Sincerely,</div><div><br></div><div>Hyungjun Lee</div><div>UT Austin.</div><div><br></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 26, 2021 at 5:31 PM Jonathan Backman <<a href="mailto:jbackman@iis.ee.ethz.ch" target="_blank">jbackman@iis.ee.ethz.ch</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div>
<p>Dear Hyungjun Lee, <br>
</p>
<p>Here is the output of the serial run:</p>
<p>----- START OF OUTPUT -----</p>
<p>Time to get kmesh 0.217 (sec)<br>
*============================================================================*<br>
| MEMORY
ESTIMATE |<br>
| Maximum RAM allocated during each phase of the
calculation |<br>
*============================================================================*<br>
| Disentanglement 34494.77
Mb |<br>
| Wannierise: 30250.47
Mb |<br>
| plot_wannier: 30250.47
Mb |<br>
*----------------------------------------------------------------------------*<br>
<br>
Starting a new Wannier90 calculation ...<br>
<br>
<br>
Reading overlaps from wannier90.mmn : File generated by VASP:
unknown system<br>
<br>
Reading projections from wannier90.amn : Projections from Vasp,
concatenated by Python.<br>
<br>
Time to read overlaps 2195.901 (sec)</p>
<p>----- END OF OUTPUT -----</p>
<p>Best, <br>
</p>
<p>Jonathan <br>
</p>
<p><br>
</p>
<div>On 26/02/2021 21:35, H. Lee wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Dear Jonathan Backman:<br>
<div><br>
</div>
<div>Thank you for providing your output.</div>
<div><br>
</div>
<div>I think that even if your run stopped during the
printing of information on b vectors, you would encounter the
issue in the next steps.</div>
<div>In normal run, the W90 output is like the following:</div>
<div><br>
</div>
<div>----- START OF EXAMPLE OUTPUT -----<br>
</div>
<div>...</div>
<div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>+----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| The
b-vectors are chosen automatically <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| The
following shells are used: <span> </span>1,<span> </span>2 <span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>+----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span>
</span>Shell <span> </span>#
Nearest-Neighbours<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span>
</span>----- <span> </span>--------------------<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span>
</span>1 <span> </span>2
<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span>
</span>2 <span> </span>6
<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>+----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|
Completeness relation is fully satisfied [Eq. (B1), PRB
56, 12847 (1997)]<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>+----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span> </span>b_k
Vectors (Ang^-1) and Weights (Ang^2)<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span> </span>----------------------------------------<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span> </span>No.
<span> </span>b_k(x)<span> </span>b_k(y)<span> </span>b_k(z)<span> </span>w_b
<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span> </span>---<span> </span>--------------------------------
<span> </span>--------<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>1
<span> </span>0.000000<span> </span>0.000000<span> </span>0.079153<span> </span>71.124740<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>2
<span> </span>0.000000<span> </span>0.000000
<span> </span>-0.079153<span> </span>71.124740<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>3
<span> </span>0.113136
<span> </span>-0.000000<span> </span>0.026384<span> </span>26.042079<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>4<span> </span>-0.113136<span> </span>0.000000
<span> </span>-0.026384<span> </span>26.042079<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>5<span> </span>-0.056568<span> </span>0.097979<span> </span>0.026384<span> </span>26.042079<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>6
<span> </span>0.056568
<span> </span>-0.097979
<span> </span>-0.026384<span> </span>26.042079<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>7<span> </span>-0.056568
<span> </span>-0.097979<span> </span>0.026384<span> </span>26.042079<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>8
<span> </span>0.056568<span> </span>0.097979
<span> </span>-0.026384<span> </span>26.042079<span> </span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>+----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span>
</span>b_k Directions (Ang^-1)<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span>
</span>-----------------------<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span> </span>No.
<span> </span>x
<span> </span>y
<span> </span>z
<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span> </span>---<span> </span>--------------------------------
<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>1
<span> </span>0.000000<span> </span>0.000000<span> </span>0.079153
<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>2
<span> </span>0.113136
<span> </span>-0.000000<span> </span>0.026384
<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>3<span> </span>-0.056568<span> </span>0.097979<span> </span>0.026384
<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>4<span> </span>-0.056568
<span> </span>-0.097979<span> </span>0.026384
<span>
</span>|</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>+----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><br>
</span></p>
<p style="margin:0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><span style="color:rgb(0,0,0);font-weight:normal"> </span><font color="#ff0000"><b>Time to get kmesh<span>
.....</span> (sec)</b></font></span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>*============================================================================*</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span>
</span>MEMORY ESTIMATE <span>
</span>|</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span>Maximum
RAM allocated during each phase of the calculation <span> </span>|</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>*============================================================================*</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span>
</span>Disentanglement <span> </span>9404.64
Mb<span>
</span>|</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span>
</span>Wannierise: <span> </span>4942.84
Mb<span>
</span>|</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>|<span>
</span>plot_wannier: <span> </span>4942.84
Mb<span>
</span>|</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>*----------------------------------------------------------------------------*</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>Starting a
new Wannier90 calculation ...</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>Reading
overlaps from wannier90.mmn<span> </span>: File
generated by ...</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>Reading
projections from wannier90.amn : File generated by ...</span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="margin:0px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo"><span style="font-variant-ligatures:no-common-ligatures"><span style="color:rgb(0,0,0);font-weight:normal"> </span><b><font color="#ff0000">Time to read overlaps<span>
.....</span> (sec)</font></b></span></p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0);min-height:13px"><span style="font-variant-ligatures:no-common-ligatures"></span><br>
</p>
<p style="margin:0px;font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>*-------------------------------
DISENTANGLE --------------------------------*</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">
</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span> </span>+----------------------------------------------------------------------------+</span></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)">...<br>
</p>
<div>----- END OF EXAMPLE OUTPUT -----</div>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:Menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"></span></p>
<div>You told me that your serial run proceeded to the
disentanglement step.</div>
<div>Could you let me know the (1) time to get kmesh and (2)
time to read overlaps (highlighted with red-color texts in
the above example output) from your W90 output obtained by
your serial run?</div>
<div><br>
</div>
</div>
<div>Sincerely,</div>
<div><br>
</div>
<div>Hyungjun Lee</div>
<div>UT Austin</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Feb 26, 2021 at 12:23
PM Jonathan Backman <<a href="mailto:jbackman@iis.ee.ethz.ch" target="_blank">jbackman@iis.ee.ethz.ch</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div>
<p>Dear Hyungjun Lee, <br>
</p>
<p>Thank you for the help.<br>
</p>
<p>I have attached the W90 output file from my calculation.
As you can see it stops printing output after picking
Shell. <br>
</p>
<p>When running the calculation using the serial version the
output also stops at this point for a long time. Then
after a few days it shows that it has done a few steps of
the disentanglement, this however never happens for the
parallel run. <br>
</p>
<p>Best, <br>
</p>
<p>Jonathan<br>
</p>
<p><br>
</p>
<div>On 26/02/2021 18:38, H. Lee wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Dear Jonathan Backman:
<div><br>
</div>
<div>Could you show me your Wannier90 (W90) output with
the high verbosity so that I can identify the step at
which W90 got stuck; in particular, I would like to
know whether your run passed the reading of relevant
input matrices, for instance, Mmn.</div>
<div><br>
</div>
<div>I assume that you use the disentanglement.</div>
<div>In this case, the largest array is the global
complex-valued array (attributed as SAVE and PUBLIC)
of m_matrix_orig and it is only allocated on the ROOT
node with the size (in your case) of 16 x 2688 x 2688
x 8 x 27 = about 25 GB (I assume nntot is 8 in your
case).</div>
<div><br>
</div>
<div>One of the issues in the current implementation is
that even if this matrix is not used any more outside
of the subroutine of overlap_read when gamma_only is
false, it is not deallocated immediately after reading
Mmn and scattering it in the subroutine of
overlap_read, thereby leading to the very large
(unbalanced) memory footprint on the ROOT node.</div>
<div><br>
</div>
<div>I understand that in your case, memory might not be
a problem, but I would like to confirm it by looking
at your W90 output.</div>
<div><br>
</div>
<div>Sincerely,</div>
<div><br>
</div>
<div>Hyungjun Lee</div>
<div>UT Austin</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Fri, Feb 26, 2021
at 4:22 AM Jonathan Backman <<a href="mailto:jbackman@iis.ee.ethz.ch" target="_blank">jbackman@iis.ee.ethz.ch</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Dear All,<br>
<br>
I'm trying to run Wannier90 using MPI for a large
system.<br>
<br>
2688 Bloch states, 2048 Wannier functions, and 27
K-points. (3x3x3 grid).<br>
<br>
AMN file size: 6 GB<br>
<br>
MMN file size: 42 GB<br>
<br>
My system does not run out of memory during the
parallel run (512 GB <br>
available).<br>
<br>
When using one MPI process then calculation
progresses, but very slow <br>
due to the size. However, when running using multiple
MPI processes <br>
the calculations runs but does not progress at all, I
have tried waiting <br>
over 2 weeks. I tried different number of MPI
processes, but I would <br>
assume 27 would be the best since I have 27 k-points.<br>
<br>
Does anyone have experience with the MPI version of
the code for large <br>
systems? Are there any specific setting that should be
used when running <br>
using MPI?<br>
<br>
Best regards,<br>
<br>
Jonathan Backman, ETH Zürich<br>
<br>
<br>
<br>
_______________________________________________<br>
Wannier mailing list<br>
<a href="mailto:Wannier@lists.quantum-espresso.org" target="_blank">Wannier@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/wannier" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/wannier</a><br>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote>
</div>
</blockquote></div>
</div></div>