<div dir="ltr"><div>Very interesting, and excellent questions, for which unfortunately I have no clear answer (nor has anybody else, I am afraid). <br><br>One should obtain the same numbers - within the errors due to roundoff, though - in serial, OpenMP, MPI execution, and on different machines, and with different compilers and mathematical libraries. In practice, there are invariably small differences, that sometimes do not completely disappear even if one pushes convergence thresholds to very strict limits. In addition to <a href="http://noncolin-constrain_total.in">noncolin-constrain_total.in</a>, another notable offender is <a href="http://vdw-ts.in">vdw-ts.in</a>. <br><br>This could signal a small bug, but in my experience, most of those cases can be linked to specific optimized mathematical libraries or compiler versions. As long as we can blame somebody else, not a big problem :-)<br><br></div><div>Paolo<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 26, 2016 at 12:23 PM, Nick Wilson <span dir="ltr"><<a href="mailto:nw.qeforge.5211@family-wilson.me.uk" target="_blank">nw.qeforge.5211@family-wilson.me.uk</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I’ve been testing the OpenMP build of Quantum Espresso 5.3.0 on our system using the Intel compiler and MKL and have a question about variation of energy with the number of OpenMP threads used.<br>
<br>
I ran all the plane wave tests in the test-suite directory using between 1 and 16 OpenMP threads and they all gave consistent results apart from pw_noncolin/<a href="http://noncolin-constrain_total.in" rel="noreferrer" target="_blank">noncolin-constrain_total.in</a> which showed variation in between -55.54478325 Ry and -55.54478414 Ry.<br>
<br>
I ran the test through the Intel Inspector tool but that didn’t show up any threading deadlocks or data races.<br>
<br>
I dropped the compiler optimisation to -O0 and added the “-fp-model strict” and “-fp-model source” compiler flags but that had no effect.<br>
<br>
I tried using some of the relevant environment variables (KMP_DETERMINISTIC_REDUCTION=1 and MKL_CBWR=COMPATIBLE) which also had no effect.<br>
<br>
Changing to use the internal BLAS library resolved the issue so it looks to be MKL-related. It’s present with both the GNU and Intel compilers.<br>
<br>
I dropped back to an earlier version of MKL but the effect was still present.<br>
<br>
As it was thread-related I tried linking against the sequential version of MKL but that didn’t help.<br>
<br>
So, I guess my questions are:<br>
Should the results always be invariant of the number of OpenMP threads?<br>
Is there anything unique about the <a href="http://noncolin-constrain_total.in" rel="noreferrer" target="_blank">noncolin-constrain_total.in</a> test case which would cause it to behave differently to the rest of the tests?<br>
<br>
Best regards,<br>
Nick Wilson<br>
<br>
<br>
System details:<br>
Intel Sandy Bridge E5-2650 CPU<br>
CentOS Linux release 7.2.1511<br>
MKL from Intel compiler 16.0.0<br>
GNU compiler version 4.8.5<br>
_______________________________________________<br>
Pw_forum mailing list<br>
<a href="mailto:Pw_forum@pwscf.org">Pw_forum@pwscf.org</a><br>
<a href="http://pwscf.org/mailman/listinfo/pw_forum" rel="noreferrer" target="_blank">http://pwscf.org/mailman/listinfo/pw_forum</a></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>Phone +39-0432-558216, fax +39-0432-558222<br><br></div></div></div></div></div>
</div>