[Pw_forum] Why does Ewald sum calculation need so much RAM

[Paolo Giannozzi]

On Tue, 2012-07-17 at 16:57 +0200, Thomas Gruber wrote:

> For example the the RAM usage of a band structure calculation
> increases from 114MB per core to 800MB per core (12 cores) by 
> changing "-npool 1" to "-npool 12". 

k-point parallelization (aka pools) does not distribute memory, 
so each processors has a complete copy of (almost) all arrays.
Plane-wave parallelization instead distributes most memory (but 
not all). Other parallelization levels, not always present, 
distribute more computation and more arrays. With larger and
more complex calculations becoming available, more computation
and more arrays that used to be considered as irrelevant become
bottlenecks and need to be distributed as well. 

> The Ewald sum calculation seems to be the part of the phonon
> calculation with the highest demand on RAM [...] For small 
> systems this Ewald sum calculation is no big deal, but for this 
> big system the scale up is pretty poor. With 16 cores it takes "3d 6h"

are we talking about RAM, or about CPU, or both, and in which subroutine
exactly? I am quite sure that the Ewald term used to be rather
insignificant on both counts, at least for systems of tens of atoms
(e.g. C60). 

> My problem is now that I have to use 32 cores per job, split into each 
> representations (451) and use "-npool 1" to get these jobs running and 
> it calculates the Ewald sum in each job, which actually should be 
> calculated only once per q point. So 2/3 of my cpu time is consumed to 
> calculate the Ewald sum 451 times instead of once.

it was done in this way because the Ewald part has never been a
serious problem for anybody until now.

>  I have already over 1 
> million CPUh and not even half of it is calculated. Is there a way to 
> save cpu time?

sure there is, but you need to locate exactly where too much cpu time is
spent and where too much RAM is required. Then, we can discuss about
possible solutions. 

P.
-- 
Paolo Giannozzi, IOM-Democritos and University of Udine, Italy