[Pw_forum] Memory question

[Axel Kohlmeyer]

On Tue, 14 Feb 2006, Miguel Martinez wrote:

hi,

MM> Hello everybody,
MM> 
MM> I'm running parallel espresso (v2.1.5) in an Itanium2 cluster. The OS is 
MM> Red Hat Linux, and the kernel is 2.4.20. The cluster uses mpich 1.2.6, 
MM> and the jobs are sent using PBS. Compiler is Intel v7. Most nodes use 
MM> 4Gb, though there are some with 16Gb RAM.
MM> 
MM> The thing is that, when I send a phonon multiprocessor run, while it 
MM> doesn't use a lot of memory (some 250Mb per thread), it swaps out. I've 
MM> run two proccessors in a 4 proc node and, to make sure other jobs don't 
MM> interfere, using all 4 procs for myself.

you have to take into account, that linux also uses the memory for disk
caching, so if you have a lot of disk i/o, then the buffering of it will
add 'pressure' to enlarge the pool of memory used for i/o buffering and
less used memory will be swapped out. the linux kernel memory management
of the default desktop distributions is tuned towards good interactive
performance and that will interfere with high i/o jobs.

one example, i experienced some time ago, was on a four way 4GB alpha 
machine, where a small memory (50MB) conventional SCF program using 
a 20GB integrals file would require at least 2GB for i/o buffering. we 
had two of those SCF programs running when somebody submitted a large
memory job (as there was seemingly not much memory used) and the 
machine immediately started swapping like mad and performance went
down the drain.

MM> 
MM> 1) Is it normal that, while it could empty memory from older jobs, my 
MM> phonon runs are swapping?
MM> 
MM> I personally haven't seen this on my laptop (gfortran instead of ifc)... 
MM> but I only do low cutoff and kpoints runs in my laptop.
MM> 
MM> 2) Will this affect performance badly? Or are phonon runs more 
MM> restricted by i/o speeds?
MM> 
MM> 3) This is a bit off topic. Is it OK to use -npools while using 
MM> processors in the same node or is it dumb?

well, you have to experiment. if my assertions from above apply to
your setup, then probably using a -npools option that would result
in using one pool per node, might be the best setting (make sure
that the nodes are properly listed in the machinefile, so that MPICH
can group them as you need them).

generally, parallelization via -npools should be much more efficient,
as there is much less communication, but then each pool requires the
full memory and the overlapping i/o on the nodes is another issue.
you also have to watch out, that you are using local scratch.

hope this helps,
    axel.


MM> 
MM> Thank you very much for your time,
MM> 
MM> Miguel
MM> 
MM> 
MM> 

-- 
=======================================================================
Axel Kohlmeyer   akohlmey at cmm.chem.upenn.edu   http://www.cmm.upenn.edu
   Center for Molecular Modeling   --   University of Pennsylvania
Department of Chemistry, 231 S.34th Street, Philadelphia, PA 19104-6323
tel: 1-215-898-1582,  fax: 1-215-573-6233,  office-tel: 1-215-898-5425
=======================================================================
If you make something idiot-proof, the universe creates a better idiot.