[Pw_forum] scaling on clusters with different communication types
a0kara01 at phys.ksu.edu
Sat Dec 9 17:04:40 CET 2006
Thanks Axel for your valuable reply.
I am actually looking for a quantitative comparison.
I want to know if by using the infiniband or myrinet one can push the good (quasilinear)
scaling to a higher number of CPU's. If someone did any benchmarking, I will
appreciate your input.
From: Axel Kohlmeyer [mailto:akohlmey at cmm.chem.upenn.edu]
Sent: Fri 12/8/2006 8:47 PM
To: Kara, Abdelkader
Cc: pw_forum at pwscf.org
Subject: Re: [Pw_forum] scaling on clusters with different communication types
On Fri, 8 Dec 2006, Kara, Abdelkader wrote:
AK> Dear all,
AK> I will appreciate it very much if you can share with me your experience
AK> of running pwscf on clusters with different communication hardware.
AK> I am interested in the scaling with the number of CPU's for the following 3
AK> different communication types:
AK> 1)gigabit ethernet
AK> 2) myrinet
AK> 3) InfiniBand
scaling depends a lot on the kind of jobs you intend to run.
pw.x scales almost independendly and very well across NEB
images and k-points even with gigabit ethernet. on top of
that you can parallelize across g-space, which is much more
demanding in terms of communication bandwidth and latency.
in this case scaling across gigabit is limited to a few
nodes. in-node performance is governed by available memory
bandwidth wich results in hyper-threading being conter-productive,
multi-core cpus having reduced efficiency (depending on job
size, i.e. cache efficiency) and opteron cpus due to dedicated
per-cpu memory busses scaling better than intel (xeon). only
very recent intel (woodcrest) xeon cpus have been demonstrated
to have a somewhat better performance and price/performance ratio.
please note, that these are some general trends observed from
some usage patterns that may not translate to your needs.
also presence of a per-node local scratch area or absence
impacts the performance. using a NFS filesystem for temporary
storage usually results in degraded performance.
basically, the larger your systems and the fewer k-points
you need use, the more important a fast interconnect becomes.
performance between infiniband and myrinet solutions is
more or less equivalent when compared with gigabit. thus
using older/obsolete hardware can be a bargain.
AK> Thank you very much for your input on this matter
AK> Kader Kara
AK> Physics Department
AK> University of Central Florida
AK> Pw_forum mailing list
AK> Pw_forum at pwscf.org
Axel Kohlmeyer akohlmey at cmm.chem.upenn.edu http://www.cmm.upenn.edu
Center for Molecular Modeling -- University of Pennsylvania
Department of Chemistry, 231 S.34th Street, Philadelphia, PA 19104-6323
tel: 1-215-898-1582, fax: 1-215-573-6233, office-tel: 1-215-898-5425
If you make something idiot-proof, the universe creates a better idiot.
More information about the users