[Pw_forum] ph.x: Avoiding the recalculation of the band structure in distributed phonon dispersion jobs

[Paolo Giannozzi]

Interesting: good to know that somebody is actually using 
"grid" parallelization. The problem you mention is known
(together with other of similar kind, e.g. the time taken
during a restart by the initialization). The solution is
however not straightforward: the phonon code is very
complex and must work under many different conditions.
Making it work for all cases and all tastes and in an
efficient way is quite a bit of a challenge. Note that
since you have full access to the source, the simplest
approach, even if not the most elegant, might be to 
edit the main code so that it performs only the operations 
you desire: for instance, reading the data files for the 
required wave-vector instead of recalculating it.

P.

On Mon, 2013-02-11 at 18:30 +0000, Karttunen Antti wrote:
> Dear all, 
> 
> We are using the start_q/last_q and start_irr/last_irr keywords to execute phonon dispersion jobs within a HPC grid service. The scheme works really nicely and we are able to run fairly large phonon dispersion calculations very efficiently. However it would be great to know if we could further increase the efficiency by avoiding the recalculation of the band structure at all irreps for every q.
> 
> A concrete example: We are using a 4x4x4 q-point grid to investigate the phonon dispersions of cubic silicon clathrate (FCC structure with 34 atoms in the primitive cell),requiring the calculation of 8 q-points in total. While the number of symmetry-independent q-points is rather low, the individual q-points can contain as many as 101 irreps (558 (q,irrep) calculations in total). While in "normal" phonon dispersion calculations the band structure is solved once for every q, in the distributed phonon dispersion calculations every single (q,irrep) job calculates the band structure before doing the actual phonon calculation (except q=1). So, the band structure is "re-calculated" numerous times in the distributed scheme. The overhead is not negligible: For a single (q,irrep) job at the q-points with the lowest symmetry,  the band structure calculation can typically take ~10 CPU hours of the total execution time of ~60 CPU hours (we are running the jobs in the grid with just one CPU).
> 
> For systems like this, it would be really great if we could do something like this:
> 1) Precalculate the band structure for every q (for example, for irrep=1),
> 2) Write the results of the band structure calculation to a file for every q 
> 3) For all other irreps, just read the precalculated band structure from the file.
> 
> We are already using a similar scheme to avoid the re-calculation of the dielectric constant for all q=1 irreps:
> 1) Precalculate the dielectric constant for (q=1,irrep=1)
> 2) Use data-file.1.xml with DIELECTRIC_CONSTANT and EFFECTIVE_CHARGES as the starting point for other q=1 irreps.
> 3) With recover=.true., the re-calculation of the dielectric constant is avoided
> 
> However, we have not been able to devise a similar scheme to avoid the re-calculation of the band structure for q>1. I've been reading the source code but at least based on check_initial_status.f90 it seems that reading the bands is only possible if there is a restart file available (i.e. the calculation has been interrupted).  So, while the built-in logic supports restarting "normal" phonon dispersion calculations, we haven't been able to find out a way to read the band structure into a single (q,irrep) job. 
> 
> We would really appreciate any comments or ideas on how to avoid the overhead from the band structure calculations in the above scenario.
> 
> Best wishes,
> Antti Karttunen
> 

-- 
Paolo Giannozzi, IOM-Democritos and University of Udine, Italy