[QE-users] Sub optimal performance on 32 core AMD machine

Tue Nov 17 20:06:01 CET 2020

Dear all,
Just some quick remarks:
OpenMP in QE is added on top of MPI, so with just one 32 / 24 cores, it 
is more or less useless to use OpenMP at all, this is true for almost 
any code with hybrid parallelization.

You claim, you have played around with OpenMPI/OpenMP settings but 
nothing changed, which settings?

If you don't share your system, at least the computational relevant 
parameters, like FFT grid size, number of plane wave coefficients, type 
of pseudo potentials, type of calculation etc nobody can guess what kind 
of performance you may or may not expect.
E.g. if may also that your simulations are just too small, if they run 
just for seconds on a quad core machine, nothing will change going to 
more cores.

The debug environment switch for MKL depends on the version of the MKL 
itself, more recent versions need more 'sophisticated' workarounds, but 
if you google you will find them, too.

Last point, if you don't specify exactly how your workstations are 
configured, nobody knows what you are talking about, same for compilers 
etc, they all carry a version number.

best,
Tobias

On 11/17/20 7:24 PM, Pamela Whitfield wrote:
> Michal
>
> I have a very similar use-case and looked into many of the same issues 
> when I got my Threadripper 3960X system at the beginning of the year 
> to supplement my old dual-Xeon setup. In the past few days I've been 
> revisiting compilation as I got hold of a Quadro GV100 for GPU 
> acceleration of my optimizations.
>
> Basically it seems as though code compiled for Zen2 either can't 
> handle code compiled for both MPI and OpenMP at all, or does so poorly 
> even when it runs.
> Best performance for pw.x on v6.5 (I've been playing with GIPAW and 
> there's no 6.6 compatible version yet) has been with a simple gcc 
> OpenMPI compilation without openmp threading and with about 20 MPI 
> cores on my 24 core CPU. Compiling with GCC or PGI compiler made 
> little difference, although only the more recent PGI compilers will 
> have zen2 optimization.
> I get little benefit from Intel MKL over openblas/lapack/fftw3 even 
> with the debug tweaks, etc.
> Puget Systems numbers with other programs suggest that OpenMP only 
> performs better than OpenMPI with Threadripper but I find the opposite 
> with QE.
> I did try disabling hyperthreading in the BIOS but that made no 
> difference to the performance.
>
> GPU compilation really shows the issue with MPI/OpenMP clashing. With 
> the Xeons I could compile code with MKL that would run well on a 
> Quadro K6000 while offloading to the CPU with MPI when needed. It 
> could still be a compiler issue (have to use PGI with the GPU version) 
> but it just doesn't work with the 3960X, and some things don't thread 
> well with pure OpenMP (e..g dftd3 versus dftd2) so I'll still need to 
> use separately compiled versions of 6.5 for different problems.
>
> BTW with a dual CPU system you may benefit from pinning threads to 
> particular CPUs - it works on the dual Xeon in any case. My 
> Threadripper balances the load across the cores in a pretty dynamic 
> manner and that's on a single socket.
>
> Best regards
> Pam Whitfield
>
> Independent Consultant
>
>
>
>
>
> Message: 1
> Date: Mon, 16 Nov 2020 15:19:04 +0100
> From: Michal Husak <Michal.Husak at vscht.cz <mailto:Michal.Husak at vscht.cz>>
> To: Quantum ESPRESSO users Forum <users at lists.quantum-espresso.org 
> <mailto:users at lists.quantum-espresso.org>>
> Subject: [QE-users] Sub optimal performance on 32 core AMD machine
> Message-ID: <fe59d3a8-2ace-4f66-a5c6-e83b01387c61 at cln92.vscht.cz 
> <mailto:fe59d3a8-2ace-4f66-a5c6-e83b01387c61 at cln92.vscht.cz>>
> Content-Type: text/plain; charset="UTF-8"; format=flowed
>
> I had purchased a new PC with 2x 16 core AMD EPYC processors . 64
> cores with hyper threading ...
> I was hoping my QM programs (Quantum Espresso, CASTEP) will run on the new
> system faster, than on my old 4 core i7 Intel machine (8 year old) ....
>
> To my great surprise, the opposite is almost true :-(.
> My main task is scf and geometry optimization of middle sized organic
> molecular crystals (abut 100 C,H,N per unit cell) ...
>
> I was playing with OpenMPI/OpenMP setup changes ...
> I was playing with the secret MKL_DEBUG_CPU_TYPE=5 parameter
> (responsible for slow run of Intel MKL compiled code on AMD) ...
>
> Nothing helps, the best speed is obteined when I  use only 4 cores
> (OpenMPI or OpenMP - results similar) ...
> Using 16 or 32 cores gives almost no benefit ...
> The CPU load for run on 1/4/816/32 coresponds to the nubmer of CPU
> set = they try to do something ...
>
> Any idea what I should check, try optimize ?
>
> Maybe the bottleneck is memory access, not CPU power  (I have 128
> GB  almost not used RAM) ?
>
> Michal Husak
>
> UCT Prague
>
>
>
> _______________________________________________
> Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
> users mailing list users at lists.quantum-espresso.org
> https://lists.quantum-espresso.org/mailman/listinfo/users

-- 
M.Sc. Tobias Klöffel
=======================================================
HPC (High Performance Computing) group
Erlangen Regional Computing Center(RRZE)
Friedrich-Alexander-Universität Erlangen-Nürnberg
Martensstr. 1
91058 Erlangen

Room: 1.133
Phone: +49 (0) 9131 / 85 - 20101

=======================================================

E-mail: tobias.kloeffel at fau.de

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20201117/900c06aa/attachment.html>