[Pw_forum] possible i/o bug in turbo_lanczos.x and turbo_davidson.x 5.3.0
Giuseppe Mattioli
giuseppe.mattioli at ism.cnr.it
Fri Feb 5 13:55:14 CET 2016
Dear Iurii
> It seems that this is a RAM issue.
Maybe something connected with memory allocation which is changed between 5.1.1 and 5.2.1
> I runned your test case with QE-5.2.1 on my local workstation with 8 cores and 64 Gb RAM and the Lanczos code crashed. When I changed the input of
> PWscf so that only the occupied states are computed (actually, the empty states are not need in the Lanczos calculation), which of course decreased
> RAM requirements, the code didn't crash.
the code didn't crash even on my 8 cores 16 GB RAM cluster with 5.1.1. And it is not a large benchmark. I used to run larger ones on the same node and
far larger ones on two nodes of the same machine with older versions. The problem cannot be due to the overall memory requirements, but to some
problematic memory allocation (a leak somewhere?).
> If this is indeed the reason of the problem, then it seems strange to me why the QE-5.1.1 does not have this problem. Maybe some investigation would
> be desired.
Indeed!!! Only microsoft windows requires more memory to do the same thing in newer versions!!!!
Best
Giuseppe
On Friday, February 05, 2016 10:07:14 AM Timrov Iurii wrote:
> Dear Giuseppe,
>
> It seems that this is a RAM issue.
>
> I runned your test case with QE-5.2.1 on my local workstation with 8 cores and 64 Gb RAM and the Lanczos code crashed. When I changed the input of
> PWscf so that only the occupied states are computed (actually, the empty states are not need in the Lanczos calculation), which of course decreased
> RAM requirements, the code didn't crash.
>
> In Bluegene machine you may try to optimize RAM too. Maybe you know, one can allocate all cores per node but use only a few of them which would
> allow you to increase RAM per core. Please note that with Davidson the RAM requirements are even much larger, so it is not easy to optimize the
> script for submission the jobs for large systems.
>
> In order to verify if you also have a memory issue, you may try to decrease the value of celldm(1), cutoffs etc. and see if the code does not crash.
>
> If this is indeed the reason of the problem, then it seems strange to me why the QE-5.1.1 does not have this problem. Maybe some investigation would
> be desired.
>
> HTH
>
> Best regards,
> Iurii
>
> --
> Dr. Iurii Timrov
> Postdoctoral Researcher
> École Polytechnique Fédérale de Lausanne,
> Theory and Simulation of Materials
> CH-1015 Lausanne, Switzerland
> +41 21 69 34 881
> http://people.epfl.ch/265334
>
> ________________________________________
> From: Giuseppe Mattioli <giuseppe.mattioli at ism.cnr.it>
> Sent: Thursday, February 4, 2016 5:59 PM
> To: Timrov Iurii
> Cc: pw_forum at pwscf.org
> Subject: Re: Re: [Pw_forum] possible i/o bug in turbo_lanczos.x and turbo_davidson.x 5.3.0
>
> Silent crash on bluegene with 5.2.1 (I have no time to compile 5.3.0 now. I may try tomorrow if you think it is important).
>
>
> Program turboTDDFT v.5.2.1 starts on 4Feb2016 at 17:56:55
>
> This program is part of the open-source Quantum ESPRESSO suite
> for quantum simulation of materials; please cite
> "P. Giannozzi et al., J. Phys.:Condens. Matter 21 395502 (2009);
> URL http://www.quantum-espresso.org",
> in publications or presentations arising from this work. More details at
> http://www.quantum-espresso.org/quote
>
> Parallel version (MPI & OpenMP), running on 2048 processor cores
> Number of MPI processes: 512
> Threads/MPI process: 4
> R & G space division: proc/nbgrp/npool/nimage = 512
>
> Reading data from directory:
> /gpfs/scratch/userexternal/gmattiol/test/tddft/run/tmp/././l0-5.3.0.save
>
> Info: using nr1, nr2, nr3 values from input
>
> Info: using nr1, nr2, nr3 values from input
>
> IMPORTANT: XC functional enforced from input :
> Exchange-correlation = SLA PW PBE PBE ( 1 4 3 4 0 0)
> Any further DFT definition will be discarded
> Please, verify this is what you really want
>
>
> Parallelization info
> --------------------
> sticks: dense smooth PW G-vecs: dense smooth PW
> Min 78 38 8 12054 4220 492
> Max 80 40 10 12104 4300 550
> Sum 40733 20369 5097 6186431 2186841 273425
> Tot 20367 10185 2549
>
>
> negative rho (up, down): 9.597E-02 0.000E+00
>
> Subspace diagonalization in iterative solution of the eigenvalue problem:
> scalapack distributed-memory algorithm (size of sub-group: 16* 16 procs)
>
>
> Warning: There are virtual states in the input file, trying to disregard in response calculation
>
> Ultrasoft (Vanderbilt) Pseudopotentials
>
> Normal read
>
> Gamma point algorithm
> 2016-02-04 17:57:18.063 (WARN ) [0x40000ee8d50] :7014845:ibm.runjob.client.Job: terminated by signal 6
> 2016-02-04 17:57:18.065 (WARN ) [0x40000ee8d50] :7014845:ibm.runjob.client.Job: abnormal termination by signal 6 from rank 295
>
> On Thursday, February 04, 2016 03:46:10 PM Timrov Iurii wrote:
> > Dear Giuseppe,
> >
> > As far as I understand the code crashes when it tries to write the vectors "d0psi" to the disc. First thing to do, I think, is to check that you
> > have enough space on the disc. If this is not the issue, then let's continue looking for a reason.
> >
> > You may want to look in the routine TDDFPT/src/lr_solve_e.f90 at lines 110-138 where the code writes vectors to the disc in parallel. Please make
> > sure that the "outdir" is the same in PWscf and in Lanczos/Davidson (and don't specify wfcdir). If this does not solve the problem, could you
> > report please also the output of Lanczos/Davidson (better Lanczos)?
> >
> > HTH
> >
> > Best regards,
> > Iurii Timrov
> > Post-doctoral researcher
> > THEOS - École Polytechnique Fédérale de Lausanne
> > Lausanne, Switzerland
> >
> >
> > ________________________________________
> > From: pw_forum-bounces at pwscf.org <pw_forum-bounces at pwscf.org> on behalf of Giuseppe Mattioli <giuseppe.mattioli at ism.cnr.it>
> > Sent: Thursday, February 4, 2016 11:34 AM
> > To: pw_forum at pwscf.org
> > Subject: [Pw_forum] possible i/o bug in turbo_lanczos.x and turbo_davidson.x 5.3.0
> >
> > Dear All
> > I'm having problems when performing nontrivial runs of turbo_davidson.x and turbo_lanczos.x with 5.2.1 and 5.3.0 versions of QE.
> > Let me say first that "trivial" runs (CH4 molecule with same pseudopotentials and cutoffs but a smaller 30 a.u.^3 cubic cell) work fine with all
> > the tested versions.
> > However, the input files for a nontrivial case that leads to crash should run on a decent pc in about 1 hr, so they provide a significant but not
> > huge test. *Note* that if I run the same input files with the 5.1.1 version (compiled against the very same environment) everything goes (more
> > slowly but) fine! The 5.3.0 (and 5.2.1) crashes have been reproduced on two different machines (intel 8 cores 16GB RAM, amd 32 cores 64 GB RAM),
> > so
> > they should not be considered as erratic.
> >
> > here is the pw.x run. The PPs are quite old and can be found in the online library (or provided by me on demand).
> >
> > &control
> >
> > calculation = 'scf'
> > restart_mode='from_scratch',
> > prefix='l0-5.3.0',
> > pseudo_dir = '/home/mattioligi/PP_UPF/',
> > outdir='/home/mattioligi/cocat/test_tddft/5.2.1/l0/5.3.0/run/tmp/',
> > nstep=300,
> > max_seconds=80000,
> > disk_io='low',
> > tprnfor=.true.,
> >
> > /
> > &system
> >
> > ibrav=1, celldm(1)=40.000000,
> > nat=42, ntyp=4, nbnd=75,
> > ecutwfc = 40.0,
> > ecutrho = 320.0,
> > nspin=1,
> >
> > /
> > &electrons
> >
> > diagonalization='david',
> > mixing_mode='plain'
> > mixing_beta=0.1
> > conv_thr=1.0d-8
> > electron_maxstep=100
> >
> > /
> > &ions
> > /
> >
> > ATOMIC_SPECIES
> > O 15.999 O_pbe.van.UPF
> > N 14.007 N.pbe-van_bm.UPF
> > C 12.011 C_pbe.van.UPF
> > H 1.008 H_pbe.van.UPF
> > ATOMIC_POSITIONS {angstrom}
> > C 4.815369179 12.355337788 8.111406911
> > C 5.639537337 12.072210478 7.018248617
> > C 6.373883049 10.886794669 6.974735758
> > H 5.707874252 12.778745273 6.179910928
> > C 4.734413944 11.441350355 9.166316558
> > H 4.235443595 13.287281698 8.140567718
> > C 6.304598307 9.977077773 8.041477142
> > H 7.012644682 10.659891408 6.111132336
> > C 5.477180541 10.260422385 9.138835842
> > H 4.092409998 11.653694694 10.031778418
> > H 5.418528381 9.546881383 9.971310698
> > N 7.058612774 8.759574945 8.006208499
> > C 6.384981399 7.544139013 8.340645249
> > C 6.997532612 6.588483316 9.168188787
> > C 5.084708421 7.308024697 7.864810575
> > C 6.325550737 5.410241765 9.493833204
> > H 8.006262126 6.776794433 9.557919083
> > C 4.414663626 6.134355690 8.210976959
> > H 4.597637090 8.055249046 7.224770074
> > C 5.030975670 5.176070562 9.020776666
> > H 6.819890970 4.670618768 10.138154855
> > H 3.397721512 5.964689741 7.832306200
> > H 4.503298572 4.249946635 9.284425745
> > C 8.412602212 8.773905175 7.652414992
> > C 9.197305040 9.938168667 7.841458619
> > C 9.043381168 7.634703664 7.098599788
> > C 10.533008285 9.972397555 7.486007356
> > H 8.740413757 10.828552107 8.290447985
> > C 10.383506998 7.674400214 6.758021800
> > H 8.466388332 6.717306584 6.931252215
> > C 11.175184928 8.838234071 6.927523312
> > H 11.098162573 10.894629696 7.663657304
> > H 10.849606517 6.778483121 6.322529487
> > C 12.554045113 8.768090174 6.529797787
> > C 13.538745611 9.729179498 6.474718127
> > H 12.882286114 7.769870632 6.203237321
> > C 13.338246843 11.096686263 6.810664645
> > N 13.160471613 12.223162736 7.083088078
> > C 14.914360413 9.407055683 6.034105289
> > O 15.832284936 10.221452163 5.956798921
> > O 15.091537629 8.085358800 5.710801225
> > H 16.043983143 8.016066678 5.436328923
> > K_POINTS {gamma}
> >
> > And here are the turbo_lanczos.x and turbo davidson.x input files
> >
> > lanczos
> >
> > &lr_input
> >
> > prefix="l0-5.3.0",
> > outdir='/state/partition1/mattioligi/34339',
> > wfcdir='/state/partition1/mattioligi/34339',
> > restart_step=6,
> > restart=.false.
> >
> > /
> > &lr_control
> >
> > itermax=12,
> > ipol=4,
> >
> > /
> >
> > davidson
> >
> > &lr_input
> >
> > prefix="l0-5.3.0",
> > outdir='/state/partition1/mattioligi/34340',
> > restart=.false.
> >
> > /
> > &lr_dav
> >
> > num_eign=2
> > num_init=4
> > num_basis_max=10
> > residue_conv_thr=1.0E-4
> > start=0.1
> > finish=1.5
> > step=0.0002
> > broadening=0.005
> > reference=0.2
> > p_nbnd_occ=5
> > p_nbnd_virt=5
> > poor_of_ram=.false.
> > poor_of_ram2=.false.
> >
> > /
> >
> > In both cases and on both machines the CRASH report is something like
> >
> > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> >
> > task # 1
> > from davcio : error # 20
> > error while writing from file "/state/partition1/mattioligi/34340/l0-5.3.0.d0psi.32"
> >
> > %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
> >
> > I suppose that it is some kind of I/O error, but I warmly require your opinion...:-)
> > Thank you in advance
> > Giuseppe
> >
> > ********************************************************
> > - Article premier - Les hommes naissent et demeurent
> > libres et égaux en droits. Les distinctions sociales
> > ne peuvent être fondées que sur l'utilité commune
> > - Article 2 - Le but de toute association politique
> > est la conservation des droits naturels et
> > imprescriptibles de l'homme. Ces droits sont la liberté,
> > la propriété, la sûreté et la résistance à l'oppression.
> > ********************************************************
> >
> > Giuseppe Mattioli
> > CNR - ISTITUTO DI STRUTTURA DELLA MATERIA
> > v. Salaria Km 29,300 - C.P. 10
> > I 00015 - Monterotondo Stazione (RM), Italy
> > Tel + 39 06 90672836 - Fax +39 06 90672316
> > E-mail: <giuseppe.mattioli at ism.cnr.it>
> > http://www.ism.cnr.it/english/staff/mattiolig
> > ResearcherID: F-6308-2012
> >
> > _______________________________________________
> > Pw_forum mailing list
> > Pw_forum at pwscf.org
> > http://pwscf.org/mailman/listinfo/pw_forum
>
> ********************************************************
> - Article premier - Les hommes naissent et demeurent
> libres et égaux en droits. Les distinctions sociales
> ne peuvent être fondées que sur l'utilité commune
> - Article 2 - Le but de toute association politique
> est la conservation des droits naturels et
> imprescriptibles de l'homme. Ces droits sont la liberté,
> la propriété, la sûreté et la résistance à l'oppression.
> ********************************************************
>
> Giuseppe Mattioli
> CNR - ISTITUTO DI STRUTTURA DELLA MATERIA
> v. Salaria Km 29,300 - C.P. 10
> I 00015 - Monterotondo Stazione (RM), Italy
> Tel + 39 06 90672836 - Fax +39 06 90672316
> E-mail: <giuseppe.mattioli at ism.cnr.it>
> http://www.ism.cnr.it/english/staff/mattiolig
> ResearcherID: F-6308-2012
********************************************************
- Article premier - Les hommes naissent et demeurent
libres et égaux en droits. Les distinctions sociales
ne peuvent être fondées que sur l'utilité commune
- Article 2 - Le but de toute association politique
est la conservation des droits naturels et
imprescriptibles de l'homme. Ces droits sont la liberté,
la propriété, la sûreté et la résistance à l'oppression.
********************************************************
Giuseppe Mattioli
CNR - ISTITUTO DI STRUTTURA DELLA MATERIA
v. Salaria Km 29,300 - C.P. 10
I 00015 - Monterotondo Stazione (RM), Italy
Tel + 39 06 90672836 - Fax +39 06 90672316
E-mail: <giuseppe.mattioli at ism.cnr.it>
http://www.ism.cnr.it/english/staff/mattiolig
ResearcherID: F-6308-2012
More information about the users
mailing list