<div dir="ltr"><div>Hi Coiby,<br><br>Using -ntg and -ndiag doesn't mean exactly using threaded library.<br></div><div>Those options are useful when your calculation involves over 1k processors without (-ni image parallelization)<br> and needed to be tested carefully at pw.x before performing any large rigorous calculations.<br>I don't think you need them.<br><br></div><div>What I meant is using threaded math library MKL. You need to build QE suite with openmp (check make.sys to see if the threaded MKL is linked). I recommend using at lease QE version 5.3.<br></div><div>If you have 32 nodes with 24 cores each,<br></div><div>export OMP_NUM_THREADS=4 # each MPI rank has 4 threads. You need to ensure each node gets only 6 MPI ranks.<br></div><div>Still run the following,<br></div><div>mpirun -np 192 ph.x -ni 4 -nk 3 -inp your_ph.input<br> You don't even need to redo you pw calculation if only the threads and total nodes used are increased correspondingly. <br></div><div><br>For phonon calculation, I also often get disk quota issues. Ph.x eats a lot of disk even though reduce_io=.true.<a name="reduce_io"></a> is set.<br></div><div>If you have more than 3 k-points, try to maximize your -nk option and reduce the corresponding -ni.<br>More -nk uses more disk but less -ni uses less disk. In sum, you should need less disk without compromising parallel efficiency.<br></div><div>If you still get disk issue, use less images and more threads.<br><br></div><div>Ciao,<br></div><div>Ye<br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>
Leadership Computing Facility<br>
Argonne National Laboratory</div></div></div>
<br><div class="gmail_quote">2016-05-05 8:42 GMT-05:00 Coiby Xu <span dir="ltr"><<a href="mailto:coiby.xu@gmail.com" target="_blank">coiby.xu@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Dear Dr. Luo,<br><div><br>Thank you for your detailed reply!<br><br>I'm sorry I disabled Mail delivery<b> </b>before so I didn't receive the email until I checked the mailing list archive.<br><br>I've successfully run phonon calculation without using <i>wf_collect=.true.</i> following your advise. This helps reduce the size of <b>outdir</b> from 142G to 48G.<br><br>For threaded MKL and FFT, I tested one case (-nimage 48 -npool 3 -ntg 2 -ndiag 4). To my surprise, it's marginally slower than the calculation without<i> -ntg 2 -ndiag 4</i>. In PHonon/examples/Image_example, I didn't find any useful info.<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">PH_IMAGE_COMMAND="$PARA_IMAGE_PREFIX $BIN_DIR/ph.x $PARA_IMAGE_POSTFIX"<br></blockquote><br>In the file environment_variables, no info about ntg and ndiag are given<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">PARA_POSTFIX=" -nk 1 -nd 1 -nb 1 -nt 1 "<br>PARA_IMAGE_POSTFIX="-ni 2 $PARA_POSTFIX"<br>PARA_IMAGE_PREFIX="mpirun -np 4"<br></blockquote><br></div><div>I also checked the job log for failed calculation ("Not diagonalizing because representation xx is not done"). Maybe ph.x crashes due to I/O problem (the size of outdir was 142G).<br><br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">forrtl: No such file or directory<br>forrtl: No such file or directory<br>forrtl: severe (28): CLOSE error, unit 20, file "Unknown"<br>Image PC Routine Line Source <br>ph.x 000000000088A00F Unknown Unknown Unknown<br>ph.x 0000000000517B26 buffers_mp_close_ 620 buffers.f90<br>ph.x 00000000004B85E8 close_phq_ 39 close_phq.f90<br>ph.x 00000000004B7888 clean_pw_ph_ 41 clean_pw_ph.f90<br>ph.x 000000000042E5EF do_phonon_ 126 do_phonon.f90<br>ph.x 000000000042A554 MAIN__ 78 phonon.f90<br>ph.x 000000000042A4B6 Unknown Unknown Unknown<br>libc.so.6 0000003921A1ED1D Unknown Unknown Unknown<br>ph.x 000000000042A3A9 Unknown Unknown Unknown </blockquote><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">forrtl: severe (28): CLOSE error, unit 20, file "Unknown"<br></blockquote></div><div> <br></div><div><br><div class="gmail_extra">Btw, I'm from School of Earth and Space Science of USTC.<br></div><div class="gmail_extra"><br>On Wed, May 4, 2016 at 07:41:30 CEST, Ye Luo <span dir="ltr"><<a href="mailto:coiby.xu@gmail.com" target="_blank">xw111luoye@gmail.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><pre>Hi Coiby,
"it seems to be one requirement to let ph.x and pw.x have the same number
of processors."
This is not true.
If you are using image parallelization in your phonon calculation, you need
to maintain the same amount of processes per image as your pw calculation.
In this way, wf_collect=.true. is not needed.
Here is an example. I assume you use k point parallelization (-nk).
1, mpirun -np 48 pw.x -nk 12 -inp your_pw.input
2, mpirun -np 192 ph.x -ni 4 -nk 12 -inp your_ph.input
In this step, you might notice "Not diagonalizing because representation
xx is not done" which is normal.
The code should not abort because of this.
3, After calculating all the representations belongs a given q or q-mesh.
Just add "recover = .true." in your_ph.input and run
mpirun -np 48 ph.x -nk 12 -inp your_ph.input
The dynamical matrix will be computed for that q.
If you are confident with threaded pw.x, ph.x also gets benefit from
threaded MKL and FFT and the time to solution is further reduced.
For more details, you can look into PHonon/examples/Image_example.
P.S.
Your affiliation is missing.
===================
Ye Luo, Ph.D.
Leadership Computing Facility
Argonne National Laboratory</pre><div><div class="h5"><br><br><div class="gmail_quote">On Wed, May 4, 2016 at 11:33 AM, Coiby Xu <span dir="ltr"><<a href="mailto:coiby.xu@gmail.com" target="_blank">coiby.xu@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Dear Quantum Espresso Developers and Users,<br><br><br></div>I'm running a phonon calculation parallelizing over the representations/q vectors. For my cluster, there are 24 cores per node. I want to use as many nodes as possible to speed up the calculation.<br><br></div><div>I set the number of parallelizations to be the number of nodes,<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">mpirun -np NUMBER_OF_NODESx24 ph.x -nimage NUMBER_OF_NODES <br></blockquote></div><div><div><br><br></div><div>If I only use 4 nodes (4 images), 8 nodes ( 8 images), the calculation will be finished successfully. However, more than 8 nodes, say 16 or 32 nodes, are used, each time running the calculation, such error will be given,<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">Not diagonalizing because representation xx is not done<br clear="all"></blockquote></div><div><div><br></div><div>Btw, I want to reduce I/O overhead by discarding `wf_collect` option, but the following way doesn't work (the number of processors and pools for scf calculation is the same to that in phonon calculation)<br><br></div><div><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">mpirun -np NUMBER_OF_NODESx24 pw.x <br></blockquote><br></div><div>ph.x complains,<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">Error in routine phq_readin (1):pw.x run with a different number of processors. <br>Use wf_collect=.true.<br></blockquote></div><div><br>The beginning output of pw.x,<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"> Parallel version (MPI), running on 96 processors<br> R & G space division: proc/nbgrp/npool/nimage = 96<br> Waiting for input...<br> Reading input from standard input<br></blockquote><br>and the beginning output of ph.x,<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"> Parallel version (MPI), running on 96 processors<br> path-images division: nimage = 4<br> R & G space division: proc/nbgrp/npool/nimage = 24<br></blockquote><br></div><div>Do I miss something? I know it's inefficient to let pw.x use so many processors, but it seems to be one requirement to let ph.x and pw.x have the same number of processors.<br></div><div><br></div><div>Thank you!<span><font color="#888888"><br><br></font></span></div><span><font color="#888888"><div>-- <br><div><div dir="ltr"><div><i>Best regards,</i></div><font color="#00cccc"><i>Coiby</i></font><div><font color="#00cccc"><br></font></div></div></div>
</div></font></span></div></div></div>
</blockquote></div></div></div></blockquote><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div><div dir="ltr"><div><i>Best regards,</i></div><font color="#00cccc"><i>Coiby</i></font><div><font color="#00cccc"><br></font></div></div></div>
</font></span></div></div></div>
</blockquote></div><br></div>