<div dir="ltr">Dear Milos,<div>Thank you. This definitely helps us make an informed decision.</div><div>Best,</div><div>Michal</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 10 Jul 2019 at 08:28, Milos Maric <<a href="mailto:mmaric@nvidia.com">mmaric@nvidia.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-3855164674873606540WordSection1">
<p class="MsoNormal">Dear Michal,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I run benchmarks on 2x Intel Xeon E5-2698 v4 and 2x Intel E5-2698 v4 + 4x Nvidia V100 16GB SMX2 (DGX-1V 16GB with only four GPUs enabled).<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Depending on the benchmark I normally see 6x to 11x speedup.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Best regards,<u></u><u></u></p>
<p class="MsoNormal">Miloš<span lang="SR-LATN-RS"><u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b>From:</b> users <<a href="mailto:users-bounces@lists.quantum-espresso.org" target="_blank">users-bounces@lists.quantum-espresso.org</a>>
<b>On Behalf Of </b>Michal Krompiec<br>
<b>Sent:</b> Tuesday, July 9, 2019 9:12 PM<br>
<b>To:</b> Quantum ESPRESSO users Forum <<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
<b>Subject:</b> Re: [QE-users] Bad scaling on GPU version QE<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Dear Pietro,<u></u><u></u></p>
<div>
<p class="MsoNormal">Thank you very much. Is this implementation restricted to Nvidia GPUs?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Best regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Michal<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Tue, 9 Jul 2019 at 19:14, Pietro BONFA' <<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">Dear Michal,<br>
<br>
I will hopefully be able to share some new benchmarking results in the<br>
wiki soon, but for the time being the most accurate estimate is in this<br>
presentation:<br>
<a href="https://on-demand.gputechconf.com/gtc-eu/2018/pdf/e8340-porting-quantum-espresso-to-gpu-accelerated-systems.pdf" target="_blank">https://on-demand.gputechconf.com/gtc-eu/2018/pdf/e8340-porting-quantum-espresso-to-gpu-accelerated-systems.pdf</a><br>
.<br>
<br>
In the slides above you will find a comparison between v6.1 and v6.3.<br>
Depending on the problem size and on the type of simulation, v6.1 can be<br>
substantially faster.<br>
<br>
Our plan is to close the gap with the future releases, since the<br>
difference mainly comes from partially implemented features (for<br>
example, charge density mixing, exchange and correlation, forces and<br>
stress are accelerated in v6.1 but not in v6.4). This, however, must be<br>
done with care in order to keep the GPU development sustainable.<br>
<br>
Hope this helps,<br>
kind regards,<br>
Pietro Bonfa'<br>
<br>
<br>
<br>
On 7/8/19 10:27 PM, Michal Krompiec wrote:<br>
> Dear Pietro,<br>
> What is the typical speed up vs a cpu-only system? Is this<br>
> <a href="https://www.dcs.warwick.ac.uk/pmbs/pmbs17/PMBS17/pres/paper3.pdf" target="_blank">
https://www.dcs.warwick.ac.uk/pmbs/pmbs17/PMBS17/pres/paper3.pdf</a> still<br>
> valid? Can you share any benchmarks on V100?<br>
> Best,<br>
> Michal Krompiec<br>
> Merck KGaA, Darmstadt, Germany<br>
><br>
> On Mon, 8 Jul 2019 at 20:29, Pietro BONFA' <<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a><br>
> <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a>>> wrote:<br>
><br>
> Dear Jing,<br>
><br>
> good point, I'll add the list of accelerated features in the wiki. In<br>
> the mean time you can check page 21 of the first presentation that you<br>
> can find here: <a href="https://gitlab.com/QEF/q-e-gpu/wikis/home#performance" target="_blank">
https://gitlab.com/QEF/q-e-gpu/wikis/home#performance</a><br>
><br>
> Exact exchange will come with the next release but you can already try<br>
> it by compiling this develop branch:<br>
> <a href="https://gitlab.com/QEF/q-e-gpu/tree/gpu-exx" target="_blank">https://gitlab.com/QEF/q-e-gpu/tree/gpu-exx</a><br>
><br>
> Kind regards,<br>
> Pietro Bonfa'<br>
><br>
> On 7/8/19 8:33 PM, JING YANG wrote:<br>
> > I am working on the latest version 6.4-a1. I have followed the<br>
> > instructions on the link you provided during the installation. I<br>
> would<br>
> > like to know which part of the calculation in pw being accelerated by<br>
> > GPU. For example, is it boosting the matrix diagonalization<br>
> during scf<br>
> > cycles, or boosting the hybrid functional calculations? I am<br>
> hoping to<br>
> > GPU acceleration can boost up the calculation on the exact<br>
> exchange in<br>
> > hybrid functional calculations. Is this the case?<br>
> ><br>
> > Thanks,<br>
> > Jing<br>
> ><br>
> > On Mon, Jul 8, 2019 at 1:57 PM Pietro BONFA'<br>
> <<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a> <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a>><br>
> > <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a> <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a>>>> wrote:<br>
> ><br>
> > Dear Jing,<br>
> ><br>
> > which GPU-enabled version are you using?<br>
> > GPU support starting from v6.1 is only available when using PGI<br>
> > compilers (well, in principle, any compiler implementing CUDA<br>
> Fortran).<br>
> ><br>
> > You can find additional information here:<br>
> ><br>
> > <a href="https://gitlab.com/QEF/q-e-gpu/wikis/home" target="_blank">https://gitlab.com/QEF/q-e-gpu/wikis/home</a><br>
> ><br>
> > Kinds regards,<br>
> > Pietro Bonfa'<br>
> ><br>
> > On 7/8/19 5:01 PM, JING YANG wrote:<br>
> > > Hi,<br>
> > > I am trying to test the scaling of GPU version of QE<br>
> recently. I<br>
> > > found out the computational time scaling of the GPU version is<br>
> > very bad.<br>
> > > Is there any unique flags or keywords I don't know about<br>
> the GPU<br>
> > > version? I am using gcc-6.2.0, openmpi-1/8/4,<br>
> > craype-accel-nvidia35. I<br>
> > > know it is not the recommended set up, but I guess if I can<br>
> > compile it<br>
> > > successfully, then it should work normally.<br>
> > ><br>
> > > Thanks,<br>
> > > Jing<br>
> > ><br>
> > > _______________________________________________<br>
> > > Quantum ESPRESSO is supported by MaX<br>
> > (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a><br>
> <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>><br>
> > <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>>)<br>
> > > users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> > <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>>><br>
> > > <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> > ><br>
> ><br>
> ><br>
> > --<br>
> > Pietro Bonfà<br>
> > Department of Mathematical, Physical and Computer Sciences,<br>
> > University of Parma,<br>
> > Italy<br>
> ><br>
> > Firma il tuo 5 per mille all’Università di Parma e aiuta così i<br>
> > nostri studenti che vogliono realizzare un’esperienza di studio<br>
> > all’estero - Indica 00308780345 nella tua denuncia dei redditi.<br>
> > _______________________________________________<br>
> > Quantum ESPRESSO is supported by MaX<br>
> > (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a><br>
> <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>><br>
> > <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>>)<br>
> > users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> > <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>>><br>
> > <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Quantum ESPRESSO is supported by MaX<br>
> (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a><br>
> <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>>)<br>
> > users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> > <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> ><br>
><br>
><br>
> --<br>
> Pietro Bonfà<br>
> Department of Mathematical, Physical and Computer Sciences,<br>
> University of Parma,<br>
> Italy<br>
><br>
> Firma il tuo 5 per mille all’Università di Parma e aiuta così i<br>
> nostri studenti che vogliono realizzare un’esperienza di studio<br>
> all’estero - Indica 00308780345 nella tua denuncia dei redditi.<br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX<br>
> (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a><br>
> <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>>)<br>
> users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
><br>
><br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a>)<br>
> users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
><br>
<br>
<br>
--<br>
Pietro Bonfà<br>
Department of Mathematical, Physical and Computer Sciences,<br>
University of Parma,<br>
Italy<br>
<br>
Firma il tuo 5 per mille all’Università di Parma e aiuta così i nostri studenti che vogliono realizzare un’esperienza di studio all’estero - Indica 00308780345 nella tua denuncia dei redditi.<br>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
</div>
<p>-----------------------------------------------------------------------------------<br>NVIDIA
GmbH<br>Wuerselen<br>Amtsgericht Aachen<br>HRB 8361<br>Managing Director: Karen
Theresa Burns</p>
<p>-----------------------------------------------------------------------------------<br>This
email message is for the sole use of the intended recipient(s) and may
contain<br>confidential information. Any unauthorized review, use,
disclosure or distribution<br>is prohibited. If you are not the intended
recipient, please contact the sender by<br>reply email and destroy all copies of
the original
message.<br>-----------------------------------------------------------------------------------</p>
</div>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu/quantum-espresso" rel="noreferrer" target="_blank">www.max-centre.eu/quantum-espresso</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a></blockquote></div>