<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1387296678;
mso-list-type:hybrid;
mso-list-template-ids:-712631612 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Dear Nicola,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">You can find the outputs for two benchmarks in the attachment.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<ul style="margin-top:0cm" type="disc">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Systems<o:p></o:p></li><ul style="margin-top:0cm" type="circle">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level2 lfo1">CPU<br>
2x Intel Xeon E5-2698 v4 (Broadwell)<br>
QE 6.4.1 (<a href="https://gitlab.com/QEF/q-e/tree/qe-6.4.1">https://gitlab.com/QEF/q-e/tree/qe-6.4.1</a>)<br>
Intel Compiler 2018.1<br>
Intel MPI 2018.1<br>
Intel MKL 2018.1<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level2 lfo1">GPU<br>
2x E5-2698 v4 (Broadwell) + 4x Nvidia V100 16GB SXM2 (DGX-1V 16GB with CUDA_VISIBLE_DEVICES=0,1,2,3)<br>
QE GPU 6.4.1a1 (<a href="https://gitlab.com/QEF/q-e-gpu/tree/qe-gpu-6.4.1a1">https://gitlab.com/QEF/q-e-gpu/tree/qe-gpu-6.4.1a1</a>)<br>
PGI 18.10<br>
CUDA 9.2<br>
OpenMPI 2.1.3<br>
Intel MKL 2018.1<br>
Configured with: ./configure --with-cuda=$CUDA_HOME --with-cuda-runtime=9.2 --with-cuda-cc=70 --enable-openmp --with-scalapack=no, added -D__GPU_MPI to DFLAGS in make.inc<o:p></o:p></li></ul>
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level1 lfo1">Benchmarks<o:p></o:p></li><ul style="margin-top:0cm" type="circle">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level2 lfo1">CsI<br>
<a href="https://github.com/electronic-structure/benchmarks/tree/master/CsI_3264nmm.cif/48">https://github.com/electronic-structure/benchmarks/tree/master/CsI_3264nmm.cif/48</a>, electron_maxstep reduced to 25 and removed max_seconds limitation<o:p></o:p></li><ul style="margin-top:0cm" type="square">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level3 lfo1">CPU<br>
npools = 1<br>
ndiag = 36<br>
NRANKS = 40<br>
OMP_NUM_THREADS = MKL_NUM_THREADS = 1<br>
time = 66060s<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level3 lfo1">GPU<br>
npools = 1<br>
NRANKS = 4<br>
OMP_NUM_THREADS = MKL_NUM_THREADS = 10<br>
time = 5880s<o:p></o:p></li></ul>
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level2 lfo1">AUSURF112<br>
<a href="https://github.com/romerojosh/qe-gpu-benchmarks/tree/master/AUSURF112">https://github.com/romerojosh/qe-gpu-benchmarks/tree/master/AUSURF112</a><o:p></o:p></li><ul style="margin-top:0cm" type="square">
<li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level3 lfo1">CPU<br>
npools = 1<br>
ndiag = 36<br>
NRANKS = 40<br>
OMP_NUM_THREADS = MKL_NUM_THREADS = 1<br>
time = 1060s<o:p></o:p></li><li class="MsoListParagraph" style="margin-left:0cm;mso-list:l0 level3 lfo1">GPU<br>
npools = 2<br>
NRANKS = 40<br>
OMP_NUM_THREADS = MKL_NUM_THREADS = 10<br>
time = 136s<o:p></o:p></li></ul>
</ul>
</ul>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Should you need help reproducing these results feel free to contact me.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Kind regards,<o:p></o:p></p>
<p class="MsoNormal">Miloš Marić<o:p></o:p></p>
<p class="MsoNormal">Developer Technology Engineer | NVIDIA Corporation<o:p></o:p></p>
<p class="MsoNormal"><a href="mailto:mmaric@nvidia.com">mmaric@nvidia.com</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>From:</b> users <users-bounces@lists.quantum-espresso.org>
<b>On Behalf Of </b>Marzari Nicola<br>
<b>Sent:</b> Wednesday, July 10, 2019 9:35 AM<br>
<b>To:</b> Quantum ESPRESSO users Forum <users@lists.quantum-espresso.org>; Borelli Marco <marco.borelli@epfl.ch><br>
<b>Subject:</b> Re: [QE-users] Bad scaling on GPU version QE<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Thanks! Would be great to have inputs/outputs of those benchmarks.
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Nicola<o:p></o:p></p>
<div id="AppleMailSignature">
<p class="MsoNormal">Sent from a tiny keyboard... Contact info: <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><a href="http://theossrv1.epfl.ch/Main/Contact">http://theossrv1.epfl.ch/Main/Contact</a><o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On 10 Jul 2019, at 09:28, Milos Maric <<a href="mailto:mmaric@nvidia.com">mmaric@nvidia.com</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Dear Michal,<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">I run benchmarks on 2x Intel Xeon E5-2698 v4 and 2x Intel E5-2698 v4 + 4x Nvidia V100 16GB SMX2 (DGX-1V 16GB with only four GPUs enabled).<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Depending on the benchmark I normally see 6x to 11x speedup.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Best regards,<o:p></o:p></p>
<p class="MsoNormal">Miloš<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><b>From:</b> users <<a href="mailto:users-bounces@lists.quantum-espresso.org">users-bounces@lists.quantum-espresso.org</a>>
<b>On Behalf Of </b>Michal Krompiec<br>
<b>Sent:</b> Tuesday, July 9, 2019 9:12 PM<br>
<b>To:</b> Quantum ESPRESSO users Forum <<a href="mailto:users@lists.quantum-espresso.org">users@lists.quantum-espresso.org</a>><br>
<b>Subject:</b> Re: [QE-users] Bad scaling on GPU version QE<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<p class="MsoNormal">Dear Pietro,<o:p></o:p></p>
<div>
<p class="MsoNormal">Thank you very much. Is this implementation restricted to Nvidia GPUs?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Best regards,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Michal<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On Tue, 9 Jul 2019 at 19:14, Pietro BONFA' <<a href="mailto:pietro.bonfa@unipr.it">pietro.bonfa@unipr.it</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt">
<p class="MsoNormal">Dear Michal,<br>
<br>
I will hopefully be able to share some new benchmarking results in the<br>
wiki soon, but for the time being the most accurate estimate is in this<br>
presentation:<br>
<a href="https://on-demand.gputechconf.com/gtc-eu/2018/pdf/e8340-porting-quantum-espresso-to-gpu-accelerated-systems.pdf" target="_blank">https://on-demand.gputechconf.com/gtc-eu/2018/pdf/e8340-porting-quantum-espresso-to-gpu-accelerated-systems.pdf</a><br>
.<br>
<br>
In the slides above you will find a comparison between v6.1 and v6.3.<br>
Depending on the problem size and on the type of simulation, v6.1 can be<br>
substantially faster.<br>
<br>
Our plan is to close the gap with the future releases, since the<br>
difference mainly comes from partially implemented features (for<br>
example, charge density mixing, exchange and correlation, forces and<br>
stress are accelerated in v6.1 but not in v6.4). This, however, must be<br>
done with care in order to keep the GPU development sustainable.<br>
<br>
Hope this helps,<br>
kind regards,<br>
Pietro Bonfa'<br>
<br>
<br>
<br>
On 7/8/19 10:27 PM, Michal Krompiec wrote:<br>
> Dear Pietro,<br>
> What is the typical speed up vs a cpu-only system? Is this<br>
> <a href="https://www.dcs.warwick.ac.uk/pmbs/pmbs17/PMBS17/pres/paper3.pdf" target="_blank">
https://www.dcs.warwick.ac.uk/pmbs/pmbs17/PMBS17/pres/paper3.pdf</a> still<br>
> valid? Can you share any benchmarks on V100?<br>
> Best,<br>
> Michal Krompiec<br>
> Merck KGaA, Darmstadt, Germany<br>
><br>
> On Mon, 8 Jul 2019 at 20:29, Pietro BONFA' <<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a><br>
> <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a>>> wrote:<br>
><br>
> Dear Jing,<br>
><br>
> good point, I'll add the list of accelerated features in the wiki. In<br>
> the mean time you can check page 21 of the first presentation that you<br>
> can find here: <a href="https://gitlab.com/QEF/q-e-gpu/wikis/home#performance" target="_blank">
https://gitlab.com/QEF/q-e-gpu/wikis/home#performance</a><br>
><br>
> Exact exchange will come with the next release but you can already try<br>
> it by compiling this develop branch:<br>
> <a href="https://gitlab.com/QEF/q-e-gpu/tree/gpu-exx" target="_blank">https://gitlab.com/QEF/q-e-gpu/tree/gpu-exx</a><br>
><br>
> Kind regards,<br>
> Pietro Bonfa'<br>
><br>
> On 7/8/19 8:33 PM, JING YANG wrote:<br>
> > I am working on the latest version 6.4-a1. I have followed the<br>
> > instructions on the link you provided during the installation. I<br>
> would<br>
> > like to know which part of the calculation in pw being accelerated by<br>
> > GPU. For example, is it boosting the matrix diagonalization<br>
> during scf<br>
> > cycles, or boosting the hybrid functional calculations? I am<br>
> hoping to<br>
> > GPU acceleration can boost up the calculation on the exact<br>
> exchange in<br>
> > hybrid functional calculations. Is this the case?<br>
> ><br>
> > Thanks,<br>
> > Jing<br>
> ><br>
> > On Mon, Jul 8, 2019 at 1:57 PM Pietro BONFA'<br>
> <<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a> <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a>><br>
> > <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a> <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a>>>> wrote:<br>
> ><br>
> > Dear Jing,<br>
> ><br>
> > which GPU-enabled version are you using?<br>
> > GPU support starting from v6.1 is only available when using PGI<br>
> > compilers (well, in principle, any compiler implementing CUDA<br>
> Fortran).<br>
> ><br>
> > You can find additional information here:<br>
> ><br>
> > <a href="https://gitlab.com/QEF/q-e-gpu/wikis/home" target="_blank">https://gitlab.com/QEF/q-e-gpu/wikis/home</a><br>
> ><br>
> > Kinds regards,<br>
> > Pietro Bonfa'<br>
> ><br>
> > On 7/8/19 5:01 PM, JING YANG wrote:<br>
> > > Hi,<br>
> > > I am trying to test the scaling of GPU version of QE<br>
> recently. I<br>
> > > found out the computational time scaling of the GPU version is<br>
> > very bad.<br>
> > > Is there any unique flags or keywords I don't know about<br>
> the GPU<br>
> > > version? I am using gcc-6.2.0, openmpi-1/8/4,<br>
> > craype-accel-nvidia35. I<br>
> > > know it is not the recommended set up, but I guess if I can<br>
> > compile it<br>
> > > successfully, then it should work normally.<br>
> > ><br>
> > > Thanks,<br>
> > > Jing<br>
> > ><br>
> > > _______________________________________________<br>
> > > Quantum ESPRESSO is supported by MaX<br>
> > (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a><br>
> <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>><br>
> > <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>>)<br>
> > > users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> > <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>>><br>
> > > <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> > ><br>
> ><br>
> ><br>
> > --<br>
> > Pietro Bonfà<br>
> > Department of Mathematical, Physical and Computer Sciences,<br>
> > University of Parma,<br>
> > Italy<br>
> ><br>
> > Firma il tuo 5 per mille all’Università di Parma e aiuta così i<br>
> > nostri studenti che vogliono realizzare un’esperienza di studio<br>
> > all’estero - Indica 00308780345 nella tua denuncia dei redditi.<br>
> > _______________________________________________<br>
> > Quantum ESPRESSO is supported by MaX<br>
> > (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a><br>
> <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>><br>
> > <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>>)<br>
> > users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> > <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>>><br>
> > <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Quantum ESPRESSO is supported by MaX<br>
> (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a><br>
> <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>>)<br>
> > users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> > <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> ><br>
><br>
><br>
> --<br>
> Pietro Bonfà<br>
> Department of Mathematical, Physical and Computer Sciences,<br>
> University of Parma,<br>
> Italy<br>
><br>
> Firma il tuo 5 per mille all’Università di Parma e aiuta così i<br>
> nostri studenti che vogliono realizzare un’esperienza di studio<br>
> all’estero - Indica 00308780345 nella tua denuncia dei redditi.<br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX<br>
> (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a><br>
> <<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">http://www.max-centre.eu/quantum-espresso</a>>)<br>
> users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
><br>
><br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a>)<br>
> users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">
https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
><br>
<br>
<br>
--<br>
Pietro Bonfà<br>
Department of Mathematical, Physical and Computer Sciences,<br>
University of Parma,<br>
Italy<br>
<br>
Firma il tuo 5 per mille all’Università di Parma e aiuta così i nostri studenti che vogliono realizzare un’esperienza di studio all’estero - Indica 00308780345 nella tua denuncia dei redditi.<br>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu/quantum-espresso" target="_blank">www.max-centre.eu/quantum-espresso</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">
users@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a><o:p></o:p></p>
</blockquote>
</div>
<p>-----------------------------------------------------------------------------------<br>
NVIDIA GmbH<br>
Wuerselen<br>
Amtsgericht Aachen<br>
HRB 8361<br>
Managing Director: Karen Theresa Burns<o:p></o:p></p>
<p>-----------------------------------------------------------------------------------<br>
This email message is for the sole use of the intended recipient(s) and may contain<br>
confidential information. Any unauthorized review, use, disclosure or distribution<br>
is prohibited. If you are not the intended recipient, please contact the sender by<br>
reply email and destroy all copies of the original message.<br>
-----------------------------------------------------------------------------------<o:p></o:p></p>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu/quantum-espresso">www.max-centre.eu/quantum-espresso</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org">users@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users">https://lists.quantum-espresso.org/mailman/listinfo/users</a><o:p></o:p></p>
</div>
</blockquote>
</div>
</div>
</body>
</html>