[Pw_forum] Performance problems due to files in outdir

[Henning Glawe]

On Wed, May 08, 2013 at 04:54:20PM +0200, Paolo Giannozzi wrote:
> inside QE there is actually a sort of "wrapper" (PW/src/buffers.f90)
> that can read/write to file or to RAM depending upon the value of a
> variable. It used to work for a single unit, but now it has been
> extended to an arbitrary number of units. Apparently, on modern 
> parallel machines, the less one writes to disk, the better.

This is only part of the truth. Most problems on these modern machines are
related to the number of files, not to the size of them.
In principle, the FHGFS we use in our cluster could deliver a performance of
about 2 gigabyte per second (tested via the usual benchmarking tools).
Most problems arise from modifications of directory entries (most prominent
creating or deleting a file or subdirectory). AFAIK the reason for this is
the caching strategy of the operating system and the hardware (raid
controllers).
Given the case of shared scratch storage, the 'ideal' strategy would be to
use only a few big files, coordinating access via MPIIO.
On a higher level, HDF5 could be used. It abstracts the usage of parallel
IO while providing an architecture-neutral storage format (binary format,
but metadata contains information about data endianness and floating point
data formats). 
My basic idea would be to replace IOTK by a wrapper around HDF5.

-- 
Mit freundlichen Grüßen
Henning Glawe

Max-Planck-Institut für Mikrostrukturphysik
Weinberg 2, 06120 Halle (Saale), Germany http://www.mpi-halle.de/~theory
Phone: +49-345-5582-613 Fax: +49-345-5511223  Email: glawe at mpi-halle.de