[QE-users] Bad scaling on GPU version QE
Milos Maric
mmaric at nvidia.com
Mon Jul 15 18:06:07 CEST 2019
Dear Nicola,
You can find the outputs for two benchmarks in the attachment.
* Systems
* CPU
2x Intel Xeon E5-2698 v4 (Broadwell)
QE 6.4.1 (https://gitlab.com/QEF/q-e/tree/qe-6.4.1)
Intel Compiler 2018.1
Intel MPI 2018.1
Intel MKL 2018.1
* GPU
2x E5-2698 v4 (Broadwell) + 4x Nvidia V100 16GB SXM2 (DGX-1V 16GB with CUDA_VISIBLE_DEVICES=0,1,2,3)
QE GPU 6.4.1a1 (https://gitlab.com/QEF/q-e-gpu/tree/qe-gpu-6.4.1a1)
PGI 18.10
CUDA 9.2
OpenMPI 2.1.3
Intel MKL 2018.1
Configured with: ./configure --with-cuda=$CUDA_HOME --with-cuda-runtime=9.2 --with-cuda-cc=70 --enable-openmp --with-scalapack=no, added -D__GPU_MPI to DFLAGS in make.inc
* Benchmarks
* CsI
https://github.com/electronic-structure/benchmarks/tree/master/CsI_3264nmm.cif/48, electron_maxstep reduced to 25 and removed max_seconds limitation
* CPU
npools = 1
ndiag = 36
NRANKS = 40
OMP_NUM_THREADS = MKL_NUM_THREADS = 1
time = 66060s
* GPU
npools = 1
NRANKS = 4
OMP_NUM_THREADS = MKL_NUM_THREADS = 10
time = 5880s
* AUSURF112
https://github.com/romerojosh/qe-gpu-benchmarks/tree/master/AUSURF112
* CPU
npools = 1
ndiag = 36
NRANKS = 40
OMP_NUM_THREADS = MKL_NUM_THREADS = 1
time = 1060s
* GPU
npools = 2
NRANKS = 40
OMP_NUM_THREADS = MKL_NUM_THREADS = 10
time = 136s
Should you need help reproducing these results feel free to contact me.
Kind regards,
Miloš Marić
Developer Technology Engineer | NVIDIA Corporation
mmaric at nvidia.com<mailto:mmaric at nvidia.com>
From: users <users-bounces at lists.quantum-espresso.org> On Behalf Of Marzari Nicola
Sent: Wednesday, July 10, 2019 9:35 AM
To: Quantum ESPRESSO users Forum <users at lists.quantum-espresso.org>; Borelli Marco <marco.borelli at epfl.ch>
Subject: Re: [QE-users] Bad scaling on GPU version QE
Thanks! Would be great to have inputs/outputs of those benchmarks.
Nicola
Sent from a tiny keyboard... Contact info:
http://theossrv1.epfl.ch/Main/Contact
On 10 Jul 2019, at 09:28, Milos Maric <mmaric at nvidia.com<mailto:mmaric at nvidia.com>> wrote:
Dear Michal,
I run benchmarks on 2x Intel Xeon E5-2698 v4 and 2x Intel E5-2698 v4 + 4x Nvidia V100 16GB SMX2 (DGX-1V 16GB with only four GPUs enabled).
Depending on the benchmark I normally see 6x to 11x speedup.
Best regards,
Miloš
From: users <users-bounces at lists.quantum-espresso.org<mailto:users-bounces at lists.quantum-espresso.org>> On Behalf Of Michal Krompiec
Sent: Tuesday, July 9, 2019 9:12 PM
To: Quantum ESPRESSO users Forum <users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>>
Subject: Re: [QE-users] Bad scaling on GPU version QE
Dear Pietro,
Thank you very much. Is this implementation restricted to Nvidia GPUs?
Best regards,
Michal
On Tue, 9 Jul 2019 at 19:14, Pietro BONFA' <pietro.bonfa at unipr.it<mailto:pietro.bonfa at unipr.it>> wrote:
Dear Michal,
I will hopefully be able to share some new benchmarking results in the
wiki soon, but for the time being the most accurate estimate is in this
presentation:
https://on-demand.gputechconf.com/gtc-eu/2018/pdf/e8340-porting-quantum-espresso-to-gpu-accelerated-systems.pdf
.
In the slides above you will find a comparison between v6.1 and v6.3.
Depending on the problem size and on the type of simulation, v6.1 can be
substantially faster.
Our plan is to close the gap with the future releases, since the
difference mainly comes from partially implemented features (for
example, charge density mixing, exchange and correlation, forces and
stress are accelerated in v6.1 but not in v6.4). This, however, must be
done with care in order to keep the GPU development sustainable.
Hope this helps,
kind regards,
Pietro Bonfa'
On 7/8/19 10:27 PM, Michal Krompiec wrote:
> Dear Pietro,
> What is the typical speed up vs a cpu-only system? Is this
> https://www.dcs.warwick.ac.uk/pmbs/pmbs17/PMBS17/pres/paper3.pdf still
> valid? Can you share any benchmarks on V100?
> Best,
> Michal Krompiec
> Merck KGaA, Darmstadt, Germany
>
> On Mon, 8 Jul 2019 at 20:29, Pietro BONFA' <pietro.bonfa at unipr.it<mailto:pietro.bonfa at unipr.it>
> <mailto:pietro.bonfa at unipr.it<mailto:pietro.bonfa at unipr.it>>> wrote:
>
> Dear Jing,
>
> good point, I'll add the list of accelerated features in the wiki. In
> the mean time you can check page 21 of the first presentation that you
> can find here: https://gitlab.com/QEF/q-e-gpu/wikis/home#performance
>
> Exact exchange will come with the next release but you can already try
> it by compiling this develop branch:
> https://gitlab.com/QEF/q-e-gpu/tree/gpu-exx
>
> Kind regards,
> Pietro Bonfa'
>
> On 7/8/19 8:33 PM, JING YANG wrote:
> > I am working on the latest version 6.4-a1. I have followed the
> > instructions on the link you provided during the installation. I
> would
> > like to know which part of the calculation in pw being accelerated by
> > GPU. For example, is it boosting the matrix diagonalization
> during scf
> > cycles, or boosting the hybrid functional calculations? I am
> hoping to
> > GPU acceleration can boost up the calculation on the exact
> exchange in
> > hybrid functional calculations. Is this the case?
> >
> > Thanks,
> > Jing
> >
> > On Mon, Jul 8, 2019 at 1:57 PM Pietro BONFA'
> <pietro.bonfa at unipr.it<mailto:pietro.bonfa at unipr.it> <mailto:pietro.bonfa at unipr.it<mailto:pietro.bonfa at unipr.it>>
> > <mailto:pietro.bonfa at unipr.it<mailto:pietro.bonfa at unipr.it> <mailto:pietro.bonfa at unipr.it<mailto:pietro.bonfa at unipr.it>>>> wrote:
> >
> > Dear Jing,
> >
> > which GPU-enabled version are you using?
> > GPU support starting from v6.1 is only available when using PGI
> > compilers (well, in principle, any compiler implementing CUDA
> Fortran).
> >
> > You can find additional information here:
> >
> > https://gitlab.com/QEF/q-e-gpu/wikis/home
> >
> > Kinds regards,
> > Pietro Bonfa'
> >
> > On 7/8/19 5:01 PM, JING YANG wrote:
> > > Hi,
> > > I am trying to test the scaling of GPU version of QE
> recently. I
> > > found out the computational time scaling of the GPU version is
> > very bad.
> > > Is there any unique flags or keywords I don't know about
> the GPU
> > > version? I am using gcc-6.2.0, openmpi-1/8/4,
> > craype-accel-nvidia35. I
> > > know it is not the recommended set up, but I guess if I can
> > compile it
> > > successfully, then it should work normally.
> > >
> > > Thanks,
> > > Jing
> > >
> > > _______________________________________________
> > > Quantum ESPRESSO is supported by MaX
> > (www.max-centre.eu/quantum-espresso<http://www.max-centre.eu/quantum-espresso>
> <http://www.max-centre.eu/quantum-espresso>
> > <http://www.max-centre.eu/quantum-espresso>)
> > > users mailing list users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
> <mailto:users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>>
> > <mailto:users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
> <mailto:users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>>>
> > > https://lists.quantum-espresso.org/mailman/listinfo/users
> > >
> >
> >
> > --
> > Pietro Bonfà
> > Department of Mathematical, Physical and Computer Sciences,
> > University of Parma,
> > Italy
> >
> > Firma il tuo 5 per mille all’Università di Parma e aiuta così i
> > nostri studenti che vogliono realizzare un’esperienza di studio
> > all’estero - Indica 00308780345 nella tua denuncia dei redditi.
> > _______________________________________________
> > Quantum ESPRESSO is supported by MaX
> > (www.max-centre.eu/quantum-espresso<http://www.max-centre.eu/quantum-espresso>
> <http://www.max-centre.eu/quantum-espresso>
> > <http://www.max-centre.eu/quantum-espresso>)
> > users mailing list users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
> <mailto:users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>>
> > <mailto:users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
> <mailto:users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>>>
> > https://lists.quantum-espresso.org/mailman/listinfo/users
> >
> >
> > _______________________________________________
> > Quantum ESPRESSO is supported by MaX
> (www.max-centre.eu/quantum-espresso<http://www.max-centre.eu/quantum-espresso>
> <http://www.max-centre.eu/quantum-espresso>)
> > users mailing list users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
> <mailto:users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>>
> > https://lists.quantum-espresso.org/mailman/listinfo/users
> >
>
>
> --
> Pietro Bonfà
> Department of Mathematical, Physical and Computer Sciences,
> University of Parma,
> Italy
>
> Firma il tuo 5 per mille all’Università di Parma e aiuta così i
> nostri studenti che vogliono realizzare un’esperienza di studio
> all’estero - Indica 00308780345 nella tua denuncia dei redditi.
> _______________________________________________
> Quantum ESPRESSO is supported by MaX
> (www.max-centre.eu/quantum-espresso<http://www.max-centre.eu/quantum-espresso>
> <http://www.max-centre.eu/quantum-espresso>)
> users mailing list users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
> <mailto:users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>>
> https://lists.quantum-espresso.org/mailman/listinfo/users
>
>
> _______________________________________________
> Quantum ESPRESSO is supported by MaX (www.max-centre.eu/quantum-espresso<http://www.max-centre.eu/quantum-espresso>)
> users mailing list users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
> https://lists.quantum-espresso.org/mailman/listinfo/users
>
--
Pietro Bonfà
Department of Mathematical, Physical and Computer Sciences,
University of Parma,
Italy
Firma il tuo 5 per mille all’Università di Parma e aiuta così i nostri studenti che vogliono realizzare un’esperienza di studio all’estero - Indica 00308780345 nella tua denuncia dei redditi.
_______________________________________________
Quantum ESPRESSO is supported by MaX (www.max-centre.eu/quantum-espresso<http://www.max-centre.eu/quantum-espresso>)
users mailing list users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
https://lists.quantum-espresso.org/mailman/listinfo/users
-----------------------------------------------------------------------------------
NVIDIA GmbH
Wuerselen
Amtsgericht Aachen
HRB 8361
Managing Director: Karen Theresa Burns
-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
_______________________________________________
Quantum ESPRESSO is supported by MaX (www.max-centre.eu/quantum-espresso<http://www.max-centre.eu/quantum-espresso>)
users mailing list users at lists.quantum-espresso.org<mailto:users at lists.quantum-espresso.org>
https://lists.quantum-espresso.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20190715/51161f7a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: benchmarks.tar.gz
Type: application/x-gzip
Size: 41173 bytes
Desc: benchmarks.tar.gz
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20190715/51161f7a/attachment.bin>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: README.txt
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20190715/51161f7a/attachment.txt>
More information about the users
mailing list