[Wannier] MPI version large systems

[Jackson, Jerome (STFC,DL,SC)]

Dear Jonathan,

I did some test calculation with 1685 bands and 1296 Wannier functions, so only half the size of yours, but "ballpark"-ish.  In that case the MMN and AMN were much smaller (intentionally), at 1.2G and 109M; with 1 k-point (but not using the gamma-only mode) it took about 18h in serial on some ordinary xeon machine.  I think wannierisation scales more or less like ~ n_wann^2 (and disentanglement n_wann^3).  So it should be quite a bit faster than you seem to be seeing.

Do you know how much memory each MPI process is using?  How many MPI processes are you running on the 512G machine?  Could it be that memory is after all being exhausted?
Yours,
Jerome Jackson.

-- 
STFC Scientific Computing Department
Daresbury Laboratory
Warrington, WA4 4AD

Tel. +44 1925 603782

> On 26 Feb 2021, at 10:22, Jonathan Backman <jbackman at iis.ee.ethz.ch> wrote:
> 
> Dear All,
> 
> I'm trying to run Wannier90 using MPI for a large system.
> 
> 2688 Bloch states, 2048 Wannier functions, and 27 K-points. (3x3x3 grid).
> 
> AMN file size: 6 GB
> 
> MMN file size: 42 GB
> 
> My system does not run out of memory during the parallel run (512 GB available).
> 
> When using one MPI process then calculation progresses, but very slow due to the size.  However, when running using multiple MPI processes  the calculations runs but does not progress at all, I have tried waiting over 2 weeks.  I tried different number of MPI processes, but I would assume 27 would be the best since I have 27 k-points.
> 
> Does anyone have experience with the MPI version of the code for large systems? Are there any specific setting that should be used when running using MPI?
> 
> Best regards,
> 
> Jonathan Backman, ETH Zürich
> 
> 
> 
> _______________________________________________
> Wannier mailing list
> Wannier at lists.quantum-espresso.org
> https://lists.quantum-espresso.org/mailman/listinfo/wannier


This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.