[QE-users] MD runs out of memory with increasing number of cores

Wed Jun 23 21:23:00 CEST 2021

Dear Prof. Giannozzi,

ah, I understand, that makes sense. Do you have any advice on how to best
track such a memory leak down in this case? The behavior is reproducible
with my setup.

Kind regards
Lenz

Am Mi., 23. Juni 2021 um 14:02 Uhr schrieb Paolo Giannozzi <
p.giannozzi at gmail.com>:

> On Wed, Jun 23, 2021 at 1:31 PM Lenz Fiedler <fiedler.lenz at gmail.com>
> wrote:
>
>  (and the increase in number of processes is most likely the reason for
>> the error, if I understand you correctly?)
>>
>
> not exactly. Too many processes may result in too much global memory
> usage, because some arrays are replicated on each process.  If you exceed
> the global available memory, the code will crash. BUT: it will do so during
> the first MD step, not after 2000 MD steps. The memory usage should not
> increase with the number of MD time steps. If it does, there is a memory
> leak, either in the code or somewhere else (libraries etc).
>
> Paolo
>
> this is not a problem. For my Beryllium calculation it is more problematic
>> since the 144 processors case really gives the best performance (I have
>> uploaded a file called performance_Be128.png to show my timing results),
>> but I still run out of memory after 2700 time steps. Although this is also
>> manageable, since I can always restart the calculation and perform another
>> 2700 time steps. With this I was able to perform 10.000 time steps in just
>> over a day. I am running more calculations on larger Be and Fe cells and I
>> will investigate this behavior there.
>>
>> I have also used the "gamma" option for the K-points to use the
>> performance benefits you outlined. For the Fe128 cell, I achieved optimal
>> performance with 144 processors and using the "gamma" option (resulting in
>> about 90s per SCF cycle). I am still not within my personal target of ~30s
>> per SCF cycle but I will start looking into the choice of my PSP and cutoff
>> (along with considering OpenMP and task group parallelization) rather than
>> blindly throwing more and more processors at the problem.
>>
>> Kind regards
>> Lenz
>>
>> PhD Student (HZDR / CASUS)
>>
>>
>> Am Sa., 19. Juni 2021 um 09:25 Uhr schrieb Paolo Giannozzi <
>> p.giannozzi at gmail.com>:
>>
>>> I tried your Fe job on a 36-core machine (with Gamma point to save time
>>> and memory) and found no evidence of memory leaks after more than 100 steps.
>>>
>>> The best performance I was able to achieve so far was with 144 cores
>>>> defaulting to -nb 144, so am I correct to assume that I should try e.g. -nb
>>>> 144 -ntg 2 for 288 cores?
>>>>
>>>
>>> You should not use option -nb except in some rather special cases.
>>>
>>> Paolo
>>>
>>>
>>> PhD Student (HZDR / CASUS)
>>>>
>>>> Am Mi., 16. Juni 2021 um 07:33 Uhr schrieb Paolo Giannozzi <
>>>> p.giannozzi at gmail.com>:
>>>>
>>>>> Hard to say without knowing exactly what goes out of which memory
>>>>> limits. Note that not all arrays are distributed across processors, so a
>>>>> considerable number of arrays are replicated on all processes. As a
>>>>> consequence the total amount of required memory will increase with the
>>>>> number of mpi processes. Also note that a 128-atom cell is not "large" and
>>>>> 144 cores are not "a small number of processors". You will not get any
>>>>> advantage by just increasing the number of processors any more, quite the
>>>>> opposite. If you have too many idle cores, you should consider
>>>>> - "task group" parallelization (option -ntg)
>>>>> - MPI+OpenMP parallelization (configure --enable-openmp)
>>>>> Please also note that ecutwfc=80 Ry is a rather large cutoff for a
>>>>> USPP (while ecutrho=320 is fine) and that running with K_POINTS Gamma
>>>>> instead of 1 1 1 0 0 0 will be faster and take less memory.
>>>>>
>>>>> Paolo
>>>>>
>>>>> On Mon, Jun 14, 2021 at 4:22 PM Lenz Fiedler <fiedler.lenz at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Dear users,
>>>>>>
>>>>>> I am trying to perform a MD simulation for a large cell (128 Fe
>>>>>> atoms, gamma point) using pw.x and I get a strange scaling behavior. To
>>>>>> test the performance I ran the same MD simulation with an increasing number
>>>>>> of nodes (2, 4, 6, 8, etc.) using 24 cores per node. The simulation is
>>>>>> successful when using 2, 4, and 6 nodes, so 48, 96 and 144 cores resp
>>>>>> (albeit slow, which is within my expectations for such a small number of
>>>>>> processors).
>>>>>> Going to 8 and more nodes, I run into an out-of-memory error after
>>>>>> about two time steps.
>>>>>> I am a little bit confused as to what could be the reason. Since a
>>>>>> smaller amount of cores works I would not expect a higher number of cores
>>>>>> to run without an oom error as well.
>>>>>> The 8 node run explictly outputs at the beginning:
>>>>>> "     Estimated max dynamical RAM per process >     140.54 MB
>>>>>>       Estimated total dynamical RAM >      26.35 GB
>>>>>> "
>>>>>>
>>>>>> which is well within the 2.5 GB I have allocated for each core.
>>>>>> I am obviously doing something wrong, could anyone point to what it
>>>>>> is?
>>>>>> The input files for a 6 and 8 node run can be found here:
>>>>>> https://drive.google.com/drive/folders/1kro3ooa2OngvddB8RL-6Iyvdc07xADNJ?usp=sharing
>>>>>> I am using QE6.6.
>>>>>>
>>>>>> Kind regards
>>>>>> Lenz
>>>>>>
>>>>>> PhD Student (HZDR / CASUS)
>>>>>> _______________________________________________
>>>>>> Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
>>>>>> users mailing list users at lists.quantum-espresso.org
>>>>>> https://lists.quantum-espresso.org/mailman/listinfo/users
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,
>>>>> Univ. Udine, via delle Scienze 206, 33100 Udine, Italy
>>>>> Phone +39-0432-558216, fax +39-0432-558222
>>>>>
>>>>> _______________________________________________
>>>>> Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
>>>>> users mailing list users at lists.quantum-espresso.org
>>>>> https://lists.quantum-espresso.org/mailman/listinfo/users
>>>>
>>>> _______________________________________________
>>>> Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
>>>> users mailing list users at lists.quantum-espresso.org
>>>> https://lists.quantum-espresso.org/mailman/listinfo/users
>>>
>>>
>>>
>>> --
>>> Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,
>>> Univ. Udine, via delle Scienze 206, 33100 Udine, Italy
>>> Phone +39-0432-558216, fax +39-0432-558222
>>>
>>> _______________________________________________
>>> Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
>>> users mailing list users at lists.quantum-espresso.org
>>> https://lists.quantum-espresso.org/mailman/listinfo/users
>>
>> _______________________________________________
>> Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
>> users mailing list users at lists.quantum-espresso.org
>> https://lists.quantum-espresso.org/mailman/listinfo/users
>
>
>
> --
> Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,
> Univ. Udine, via delle Scienze 206, 33100 Udine, Italy
> Phone +39-0432-558216, fax +39-0432-558222
>
> _______________________________________________
> Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
> users mailing list users at lists.quantum-espresso.org
> https://lists.quantum-espresso.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20210623/e521b2b9/attachment.html>