<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>Dear Mohammad,</p>
<p><br>
</p>
<p><font size="2"><span style="font-size:10pt">> 2. You are using DFT+U, more precisely<br>
> U_projection_type = 'ortho-atomic'<br>
> The GPU acceleration of this functionality is limited and some portions <br>
> of the algorithm will still run on the CPU. In addition, I fear that the <br>
> evaluation of forces with that projection method scales pretty bad with <br>
> the number of atoms, but I let the experts (and developers) of this new <br>
> functionality further comment this last point.</span></font><br>
</p>
<p><br>
</p>
<p>Yes, indeed, in DFT+U with the 'ortho-atomic' Hubbard manifold the calculations of Hubbard forces and stress take much more time than with the 'atomic' Hubbard manifold. See Phys. Rev. B 102, 235159 (2020), in particular see Appendix C.</p>
<p><br>
</p>
<p>As Pietro said, the ortho-atomic Hubbard forces and stress were ported to the GPU version of QE but this is still not optimal and can be improved in the future.</p>
<p><br>
</p>
<p>Greetings,</p>
<p>Iurii<br>
</p>
<p><br>
</p>
<div id="x_Signature">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:rgb(0,0,0); font-family:Calibri,Helvetica,sans-serif,"EmojiFont","Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols">
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<font size="3" face="'Times New Roman', Times, serif" color="808080">--<br>
<span style="font-family:Cambria,Georgia,serif">Dr. Iurii TIMROV</span><br>
<span style="font-family:Cambria,Georgia,serif">Postdoctoral Researcher</span><br>
<span style="font-family:Cambria,Georgia,serif"></span><font color="808080"><font face="'Times New Roman', Times, serif"></font></font></font></div>
<span style="font-family:Cambria,Georgia,serif"></span><font color="808080"></font><span style="font-family:Cambria,Georgia,serif"></span>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<span style="font-family:Cambria,Georgia,serif"></span><font size="3" face="'Times New Roman', Times, serif" color="808080"><span style="font-family:Cambria,Georgia,serif">STI - IMX
</span><font color="808080"><span style="font-family:Cambria,Georgia,serif"></span><font face="'Times New Roman', Times, serif"><span style="font-family:Cambria,Georgia,serif">- THEOS</span></font></font></font><font size="3" face="'Times New Roman', Times, serif" color="808080"><span style="font-family:Cambria,Georgia,serif">
and NCCR - MARVEL</span><br>
<span style="font-family:Cambria,Georgia,serif"></span></font></div>
<span style="font-family:Cambria,Georgia,serif"></span>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<span style="font-family:Cambria,Georgia,serif"></span><font size="3" face="'Times New Roman', Times, serif" color="808080"><font size="3" face="'Times New Roman', Times, serif" color="808080"><span style="font-family:Cambria,Georgia,serif">Swiss Federal Institute
of Technology Lausanne (EPFL</span><font color="808080"><font face="'Times New Roman', Times, serif"><span style="font-family:Cambria,Georgia,serif">)</span></font></font></font><br>
<span style="font-family:Cambria,Georgia,serif"></span></font></div>
<span style="font-family:Cambria,Georgia,serif"></span><font color="808080"></font><span style="font-family:Cambria,Georgia,serif"></span>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<span style="font-family:Cambria,Georgia,serif"></span><font size="3" face="'Times New Roman', Times, serif" color="808080"><span style="font-family:Cambria,Georgia,serif">CH-1015 Lausanne, Switzerland</span><br>
<span style="font-family:Cambria,Georgia,serif">+41 21 69 34 881</span></font></div>
<span style="font-family:Cambria,Georgia,serif"></span>
<div name="x_divtagdefaultwrapper" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:; margin:0">
<span style="font-family:Cambria,Georgia,serif"></span><a href="http://people.epfl.ch/265334" tabindex="0" id="LPNoLP"><span style="font-family:Cambria,Georgia,serif">http://people.epfl.ch/265334</span></a><br>
</div>
</div>
</div>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> users <users-bounces@lists.quantum-espresso.org> on behalf of Pietro Bonfa' <pietro.bonfa@unipr.it><br>
<b>Sent:</b> Monday, January 4, 2021 11:52:31 AM<br>
<b>To:</b> users@lists.quantum-espresso.org<br>
<b>Subject:</b> Re: [QE-users] [QE-GPU] computing force and stress - time cost</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">Dear Mohammad,<br>
<br>
Paolo is right, but a couple of comments can already be made:<br>
<br>
1. you are not using OpenMP parallelism, and I believe you have more <br>
than 2 cores in your system. In order to achieve a decent speedup (at <br>
least in the SCF) it's mandatory to enable openmp and exploit the whole <br>
CPU power.<br>
<br>
2. You are using DFT+U, more precisely<br>
<br>
U_projection_type = 'ortho-atomic'<br>
<br>
The GPU acceleration of this functionality is limited and some portions <br>
of the algorithm will still run on the CPU. In addition, I fear that the <br>
evaluation of forces with that projection method scales pretty bad with <br>
the number of atoms, but I let the experts (and developers) of this new <br>
functionality further comment this last point.<br>
<br>
Best regards and happy new year,<br>
Pietro<br>
<br>
<br>
<br>
<br>
On 1/4/21 10:07 AM, Paolo Giannozzi wrote:<br>
> The most important piece of information (the final time report) is not <br>
> contained in your 45Mb output.<br>
> <br>
> Paolo<br>
> <br>
> On Mon, Jan 4, 2021 at 6:33 AM Mohammad Moaddeli <br>
> <mohammad.moaddeli@gmail.com <<a href="mailto:mohammad.moaddeli@gmail.com">mailto:mohammad.moaddeli@gmail.com</a>>> wrote:<br>
> <br>
> Dear Pietro,<br>
> <br>
> It takes about 22 hours to perform an scf (about 2 hours to perform<br>
> diagonalization until convergence is achieved, and about 20 hours to<br>
> compute force and stress).<br>
> Here is the google drive link containing input and output files:<br>
> <a href="https://drive.google.com/file/d/1DFtLqFvrc8CFo1_q_jjnMFErvXpWEAHB/view?usp=sharing">
https://drive.google.com/file/d/1DFtLqFvrc8CFo1_q_jjnMFErvXpWEAHB/view?usp=sharing</a><br>
> <<a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1DFtLqFvrc8CFo1_q_jjnMFErvXpWEAHB%2Fview%3Fusp%3Dsharing&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257890985%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=s5PSyKLXmuHXgzPSj9xcsCqPXt%2BmKPw9mj%2BZACc9sSk%3D&reserved=0">https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdrive.google.com%2Ffile%2Fd%2F1DFtLqFvrc8CFo1_q_jjnMFErvXpWEAHB%2Fview%3Fusp%3Dsharing&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257890985%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=s5PSyKLXmuHXgzPSj9xcsCqPXt%2BmKPw9mj%2BZACc9sSk%3D&reserved=0</a>><br>
> <br>
> hp.x will be performed after vc-relax is done.<br>
> <br>
> Thanks in advance,<br>
> Mohammad<br>
> <br>
> On Mon, Jan 4, 2021 at 2:38 AM Pietro Bonfa' <pietro.bonfa@unipr.it<br>
> <<a href="mailto:pietro.bonfa@unipr.it">mailto:pietro.bonfa@unipr.it</a>>> wrote:<br>
> <br>
> Dear Mohammad,<br>
> <br>
> the performance of the GPU code depends dramatically on the<br>
> portions of<br>
> computation that are still performed on the CPU. Only a portion<br>
> of all<br>
> contributions to forces have been accelerated, and what is left<br>
> out may<br>
> be optimized for MPI parallelism rather than openmp.<br>
> <br>
> That being said, the behavior that you report is definitively<br>
> unusual.<br>
> Would you mind sharing input and output files?<br>
> <br>
> Best regards,<br>
> Pietro<br>
> <br>
> <br>
> <br>
> On 1/3/21 8:52 AM, Mohammad Moaddeli wrote:<br>
> > Dear all,<br>
> ><br>
> > GPU enabled QE v.6.7 is compiled on a VOLTA card. I am trying<br>
> to run a<br>
> > vc-relax for a bulk containing 48 atoms. Although<br>
> diagonalization<br>
> > (davidson) is about 3x faster than CPU, it takes a lot of<br>
> time (a couple<br>
> > of hours) to compute force and stress. Is this something<br>
> related to the<br>
> > code itself?<br>
> ><br>
> > Best,<br>
> ><br>
> > Mohammad Moaddeli<br>
> > ShirazU<br>
> ><br>
> > _______________________________________________<br>
> > Quantum ESPRESSO is supported by MaX (<a href="http://"></a>www.max-centre.eu<br>
> <<a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257900986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=ayiwKOVdIHx6hQ4VpfrjS1rTXOEiHF3ZWgGTRVbawCo%3D&reserved=0">https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257900986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=ayiwKOVdIHx6hQ4VpfrjS1rTXOEiHF3ZWgGTRVbawCo%3D&reserved=0</a>>)<br>
> > users mailing list users@lists.quantum-espresso.org<br>
> <<a href="mailto:users@lists.quantum-espresso.org">mailto:users@lists.quantum-espresso.org</a>><br>
> > <a href="https://lists.quantum-espresso.org/mailman/listinfo/users">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> <<a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257900986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=nnTzluNl%2FpWGpQ3HPy0Zm%2Bjvt4qUlNIW5mNyncKQWoU%3D&reserved=0">https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257900986%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=nnTzluNl%2FpWGpQ3HPy0Zm%2Bjvt4qUlNIW5mNyncKQWoU%3D&reserved=0</a>><br>
> ><br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX (<a href="http://"></a>www.max-centre.eu<br>
> <<a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257910979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=VDmrELj9IT2pHITCuC6xEAj%2Bc9PJuhhij7noi0qPqsM%3D&reserved=0">https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257910979%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=VDmrELj9IT2pHITCuC6xEAj%2Bc9PJuhhij7noi0qPqsM%3D&reserved=0</a>>)<br>
> users mailing list users@lists.quantum-espresso.org<br>
> <<a href="mailto:users@lists.quantum-espresso.org">mailto:users@lists.quantum-espresso.org</a>><br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> <<a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257920970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=IBKLu4F2OC%2BLtjc0eeKxBt1m1dMEzLizqXZ1EbH3%2FSI%3D&reserved=0">https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257920970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=IBKLu4F2OC%2BLtjc0eeKxBt1m1dMEzLizqXZ1EbH3%2FSI%3D&reserved=0</a>><br>
> <br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX (<a href="http://"></a>www.max-centre.eu<br>
> <<a href="https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257920970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=OYY1he5ImqA5ypHVCP0oxQLCMVoqVa3hXbngEJRmZhw%3D&reserved=0">https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.max-centre.eu%2F&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257920970%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=OYY1he5ImqA5ypHVCP0oxQLCMVoqVa3hXbngEJRmZhw%3D&reserved=0</a>>)<br>
> users mailing list users@lists.quantum-espresso.org<br>
> <<a href="mailto:users@lists.quantum-espresso.org">mailto:users@lists.quantum-espresso.org</a>><br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> <<a href="https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257930965%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=TliW5iLzHDjHe1J2j51oBqQAi2iQ8VIb12QQesVXmZA%3D&reserved=0">https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.quantum-espresso.org%2Fmailman%2Flistinfo%2Fusers&data=04%7C01%7Cpietro.bonfa%40unipr.it%7Ce34d20b789e54091c08008d8b0906452%7Cbb064bc5b7a841ecbabed7beb3faeb1c%7C0%7C0%7C637453483257930965%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&sdata=TliW5iLzHDjHe1J2j51oBqQAi2iQ8VIb12QQesVXmZA%3D&reserved=0</a>><br>
> <br>
> <br>
> <br>
> -- <br>
> Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,<br>
> Univ. Udine, via delle Scienze 208, 33100 Udine, Italy<br>
> Phone +39-0432-558216, fax +39-0432-558222<br>
> <br>
> <br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu">www.max-centre.eu</a>)<br>
> users mailing list users@lists.quantum-espresso.org<br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> <br>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu">www.max-centre.eu</a>)<br>
users mailing list users@lists.quantum-espresso.org<br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
</div>
</span></font>
</body>
</html>