<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix"><tt>Dear Filippo,<br>
dear all,<br>
<br>
on BGQ, in order to obtain the maximum performance <br>
out of the processors, one has to overload them with<br>
2/4 threads or tasks per core.<br>
In the test I've sent yesterday the number of tasks per core<br>
was 8, so that the number of threads per node turns out to be 32<br>
(running on 16 physical cores).<br>
Now I've verified that a slightly better performances can be
obtained<br>
using 8 tasks and 8 threads per task, for a total of 64 threads<br>
(again running on 16 physical core).<br>
Moreover, it seems that on this machine the band parallelism is
really effective<br>
(this is what I was guessing when I've implemented it, but I
wasn't sure...).<br>
I report some new figures about a new test with an higher degree
of overloading<br>
(4) and more band groups:<br>
Number of | Number of | sec/ | OpenMP | command line<br>
real cores | virtual cores | iteration | threads | parameters<br>
32768 | 131072 | 65 | 8 | -nbgrp 16
-ntg 4 -ndiag 512<br>
<br>
Without changing the number of physical cores, we got a 25%<br>
of improvement in the code performance (using two time more
virtual cores<br>
and two time more band groups).<br>
<br>
Well, note that, we reach the limit in the number of digits for
the<br>
processors count:<br>
Parallel version (MPI & OpenMP), running on *****
processor cores<br>
Number of MPI processes: 16384<br>
Threads/MPI process: 8<br>
band groups division: nbgrp = 16<br>
R & G space division: proc/pool = ****<br>
wavefunctions fft division: fft/group = 4<br>
<br>
Finally, consider that the total number of virtual cores on
FERMI <br>
is equal to: 655360<br>
<br>
<br>
best,<br>
carlo<br>
<br>
<br>
<br>
<br>
<br>
</tt><br>
Il 31/07/2012 06:54, Filippo Spiga ha scritto:<br>
</div>
<blockquote
cite="mid:80B1DF3C-BC43-4A2C-A663-5E2EFBD5B837@gmail.com"
type="cite">Dear Carlo,
<div><br>
</div>
<div>
<div>On Jul 30, 2012, at 6:14 PM, Carlo Cavazzoni wrote:</div>
<blockquote type="cite">
<div>Number of | Number of | sec/ | OpenMP |
command line<br>
real cores | virtual cores | iteration | threads |
parameters<br>
4096 | 8192 | 231 | 4 | -nbgrp 2
-ntg 4 <br>
-ndiag 256<br>
8192 | 16384 | 160 | 4 | -nbgrp 4
-ntg 4 <br>
-ndiag 1024<br>
16384 | 32768 | 131 | 4 | -nbgrp 4
-ntg 4 <br>
-ndiag 1024<br>
32768 | 65536 | 86 | 4 | -nbgrp 8
-ntg 4 <br>
-ndiag 2048</div>
</blockquote>
<br>
</div>
<div>benchmarking GPU PWscf on medium/big systems (>500 atoms)
I found in the PW code several spots where adding OpenMP will
improve the performance (of those sections) of a factor (at
least) 2. I haven't committed anything yet. However, it is
interesting to evaluate the OpenMP efficiency/scalability. I see
you did tests using 4 OpenMP (I assume 8 MPI per A2 chip, 2
OpenMP thread per physical core, 2 GByte per RAM each task.
correct?). What about 8 OpenMP threads? or 16? Is it worth to go
over 4 OpenMP threads?
<div>
<div><br>
</div>
<div><br>
<blockquote type="cite">
<div>(Volunteer are welcome too!)<br>
</div>
</blockquote>
</div>
<div><br>
</div>
<div>I am more than happy to help (-:</div>
<div><br>
</div>
<div>F.</div>
<br>
<div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse:
separate; color: rgb(0, 0, 0); font-family: Helvetica;
font-style: normal; font-variant: normal; font-weight:
normal; letter-spacing: normal; line-height: normal;
orphans: 2; text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-border-horizontal-spacing: 0px;
-webkit-border-vertical-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; font-size: medium; "><span class="Apple-style-span"
style="border-collapse: separate; color: rgb(0, 0, 0);
font-family: Helvetica; font-style: normal;
font-variant: normal; font-weight: normal;
letter-spacing: normal; line-height: normal; orphans: 2;
text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-border-horizontal-spacing:
0px; -webkit-border-vertical-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; font-size: medium; ">
<div style="word-wrap: break-word; -webkit-nbsp-mode:
space; -webkit-line-break: after-white-space; "><span
class="Apple-style-span" style="border-collapse:
separate; color: rgb(0, 0, 0); font-family:
Helvetica; font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: 2; text-align:
-webkit-auto; text-indent: 0px; text-transform:
none; white-space: normal; widows: 2; word-spacing:
0px; -webkit-border-horizontal-spacing: 0px;
-webkit-border-vertical-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; font-size: medium; ">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space; "><span
class="Apple-style-span" style="border-collapse:
separate; color: rgb(0, 0, 0); font-family:
Helvetica; font-style: normal; font-variant:
normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2;
text-align: -webkit-auto; text-indent: 0px;
text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px;
-webkit-border-horizontal-spacing: 0px;
-webkit-border-vertical-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; font-size:
medium; ">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space; -webkit-line-break:
after-white-space; "><span
class="Apple-style-span"
style="border-collapse: separate; color:
rgb(0, 0, 0); font-family: Helvetica;
font-style: normal; font-variant: normal;
font-weight: normal; letter-spacing: normal;
line-height: normal; orphans: 2;
text-indent: 0px; text-transform: none;
white-space: normal; widows: 2;
word-spacing: 0px;
-webkit-border-horizontal-spacing: 0px;
-webkit-border-vertical-spacing: 0px;
-webkit-text-decorations-in-effect: none;
-webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px; font-size:
medium; ">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space; "><span
class="Apple-style-span"
style="border-collapse: separate; color:
rgb(0, 0, 0); font-family: Helvetica;
font-style: normal; font-variant:
normal; font-weight: normal;
letter-spacing: normal; line-height:
normal; orphans: 2; text-indent: 0px;
text-transform: none; white-space:
normal; widows: 2; word-spacing: 0px;
-webkit-border-horizontal-spacing: 0px;
-webkit-border-vertical-spacing: 0px;
-webkit-text-decorations-in-effect:
none; -webkit-text-size-adjust: auto;
-webkit-text-stroke-width: 0px;
font-size: medium; ">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break: after-white-space;
"><span class="Apple-style-span"
style="border-collapse: separate;
color: rgb(0, 0, 0); font-family:
Helvetica; font-style: normal;
font-variant: normal; font-weight:
normal; letter-spacing: normal;
line-height: normal; orphans: 2;
text-indent: 0px; text-transform:
none; white-space: normal; widows:
2; word-spacing: 0px;
-webkit-border-horizontal-spacing:
0px;
-webkit-border-vertical-spacing:
0px;
-webkit-text-decorations-in-effect:
none; -webkit-text-size-adjust:
auto; -webkit-text-stroke-width:
0px; font-size: medium; ">
<div style="word-wrap: break-word;
-webkit-nbsp-mode: space;
-webkit-line-break:
after-white-space; "><span
class="Apple-style-span"
style="border-collapse:
separate; color: rgb(0, 0, 0);
font-family: Helvetica;
font-style: normal;
font-variant: normal;
font-weight: normal;
letter-spacing: normal;
line-height: normal; orphans: 2;
text-indent: 0px;
text-transform: none;
white-space: normal; widows: 2;
word-spacing: 0px;
-webkit-border-horizontal-spacing:
0px;
-webkit-border-vertical-spacing:
0px;
-webkit-text-decorations-in-effect:
none; -webkit-text-size-adjust:
auto; -webkit-text-stroke-width:
0px; font-size: medium; ">
<div style="word-wrap:
break-word; -webkit-nbsp-mode:
space; -webkit-line-break:
after-white-space; "><span
class="Apple-style-span"
style="border-collapse:
separate; color: rgb(0, 0,
0); font-family: Helvetica;
font-style: normal;
font-variant: normal;
font-weight: normal;
letter-spacing: normal;
line-height: normal;
orphans: 2; text-indent:
0px; text-transform: none;
white-space: normal; widows:
2; word-spacing: 0px;
-webkit-border-horizontal-spacing:
0px;
-webkit-border-vertical-spacing:
0px;
-webkit-text-decorations-in-effect:
none;
-webkit-text-size-adjust:
auto;
-webkit-text-stroke-width:
0px; font-size: medium; ">
<div style="word-wrap:
break-word;
-webkit-nbsp-mode: space;
-webkit-line-break:
after-white-space; "><span
class="Apple-style-span"
style="border-collapse:
separate; color: rgb(0,
0, 0); font-family:
Helvetica; font-style:
normal; font-variant:
normal; font-weight:
normal; letter-spacing:
normal; line-height:
normal; orphans: 2;
text-indent: 0px;
text-transform: none;
white-space: normal;
widows: 2; word-spacing:
0px;
-webkit-border-horizontal-spacing:
0px;
-webkit-border-vertical-spacing:
0px;
-webkit-text-decorations-in-effect:
none;
-webkit-text-size-adjust:
auto;
-webkit-text-stroke-width:
0px; font-size: medium;
">
<div style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space; "><span
class="Apple-style-span" style="border-collapse: separate; color: rgb(0,
0, 0); font-family:
Helvetica;
font-style: normal;
font-variant:
normal; font-weight:
normal;
letter-spacing:
normal; line-height:
normal; orphans: 2;
text-indent: 0px;
text-transform:
none; white-space:
normal; widows: 2;
word-spacing: 0px;
-webkit-border-horizontal-spacing:
0px;
-webkit-border-vertical-spacing:
0px;
-webkit-text-decorations-in-effect:
none;
-webkit-text-size-adjust:
auto;
-webkit-text-stroke-width:
0px; font-size:
medium; ">
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;
"><span
class="Apple-style-span"
style="border-collapse:
separate; color:
rgb(0, 0, 0);
font-family:
Helvetica;
font-style:
normal;
font-variant:
normal;
font-weight:
normal;
letter-spacing:
normal;
line-height:
normal; orphans:
2; text-indent:
0px;
text-transform:
none;
white-space:
normal; widows:
2; word-spacing:
0px;
-webkit-border-horizontal-spacing:
0px;
-webkit-border-vertical-spacing:
0px;
-webkit-text-decorations-in-effect:
none;
-webkit-text-size-adjust:
auto;
-webkit-text-stroke-width:
0px; font-size:
medium; ">
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;
"><span
class="Apple-style-span"
style="border-collapse:
separate;
color: rgb(0,
0, 0);
font-family:
Helvetica;
font-style:
normal;
font-variant:
normal;
font-weight:
normal;
letter-spacing:
normal;
line-height:
normal;
orphans: 2;
text-indent:
0px;
text-transform:
none;
white-space:
normal;
widows: 2;
word-spacing:
0px;
-webkit-border-horizontal-spacing:
0px;
-webkit-border-vertical-spacing:
0px;
-webkit-text-decorations-in-effect:
none;
-webkit-text-size-adjust:
auto;
-webkit-text-stroke-width:
0px;
font-size:
medium; ">
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;
"><span
class="Apple-style-span"
style="border-collapse:
separate;
color: rgb(0,
0, 0);
font-family:
Helvetica;
font-style:
normal;
font-variant:
normal;
font-weight:
normal;
letter-spacing:
normal;
line-height:
normal;
orphans: 2;
text-indent:
0px;
text-transform:
none;
white-space:
normal;
widows: 2;
word-spacing:
0px;
-webkit-border-horizontal-spacing:
0px;
-webkit-border-vertical-spacing:
0px;
-webkit-text-decorations-in-effect:
none;
-webkit-text-size-adjust:
auto;
-webkit-text-stroke-width:
0px;
font-size:
medium; ">
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;
">-- <br>
Mr. Filippo
SPIGA (穗安駒),
HPC and GPU
Technologist <spiga.filippo_at_gmail.com></div>
<div
style="word-wrap:
break-word;
-webkit-nbsp-mode:
space;
-webkit-line-break:
after-white-space;
">website: <a
moz-do-not-send="true" href="http://filippospiga.me">http://filippospiga.me</a>
~
skype: filippo.spiga<br>
<br>
«Nobody will
drive us out
of Cantor's
paradise.»
~ David
Hilbert</div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
</span></span>
</div>
<br>
</div>
</div>
</blockquote>
<br>
<br>
<pre class="moz-signature" cols="72">--
Ph.D. Carlo Cavazzoni
SuperComputing Applications and Innovation Department
CINECA - Via Magnanelli 6/3, 40033 Casalecchio di Reno (Bologna)
Tel: +39 051 6171411 Fax: +39 051 6132198
<a class="moz-txt-link-abbreviated" href="http://www.cineca.it">www.cineca.it</a></pre>
</body>
</html>