[Pw_forum] Performance problems due to files in outdir

[Axel Kohlmeyer]

On Tue, May 7, 2013 at 8:57 PM, Henning Glawe <glawe at mpi-halle.mpg.de> wrote:
> Moin,
> we have recently discovered performance problems on our parallel scratch file
> system (fhgfs) in a 400-node-cluster.
> Much of the blame went to my quantum espresso jobs (custom-patched 4.3.2). I
> am performing calculations on thousands of small crystals, most having no
> symmetry, leading to a huge number of irreducable k-points.
> At least in this version, QE/iotk creates one directory per k-point, with only
> one small file (eigenval.xml) contained in it.
> Even if I set '&system::disk_io="low"', this behaviour seems to have a major
> impact on the overall performance of the file system.

have you tried using a ramdisk? if your machine is linux based, then
you can usually access up to half of the physical memory via /dev/shm.
given the ever increasing amount of RAM that current many-core machine
have, this has become a viable alternative to modifying the source
code to keep data in core.

> Are there any plans to move away from the 'many-small-files' paradigm? Or is
> there any way/switch to make QE use another scheme for the scratch files?

whatever you do, it will require some programming effort. this can be
done from inside of QE (tedious) or on the outside by using a ramdisk
(as mentioned above) or faking it, e.g. by wrapping the stdio library
to write to huge RAM buffers for all files matching a given pattern.
cray used to provide such a library on their XT series machines to
work around the same kind of problems that you are describing on their
lustre scratch file systems.

if there are some people here on the list that would be interested in
having such a wrapper and would be volunteering some time testing and
providing feedback, i might be tempted to give it a try and write such
a wrapper.

axel.


> --
> Mit freundlichen Grüßen
> Henning Glawe
>
> Max-Planck-Institut für Mikrostrukturphysik
> Weinberg 2, 06120 Halle (Saale), Germany http://www.mpi-halle.de/~theory
> Phone: +49-345-5582-613 Fax: +49-345-5511223  Email: glawe at mpi-halle.de
> _______________________________________________
> Pw_forum mailing list
> Pw_forum at pwscf.org
> http://pwscf.org/mailman/listinfo/pw_forum



--
Dr. Axel Kohlmeyer  akohlmey at gmail.com  http://goo.gl/1wk0
International Centre for Theoretical Physics, Trieste. Italy.