<div dir="ltr"><div>Relying on compiler to move page between host and device via the PCI-E bus scares me and I'm more comfortable with an offload model which allows me to control data movement.<br></div><div>NEC recently upstreams an Aurora plugin into the OpenMP offload runtime library in LLVM.</div><div>Once the LLVM IR to NEC instruction backend is fully working (Maybe already working). We should be able to use OpenMP offload and Flang (not yet in production) which is now in LLVM.</div><div>Ye<br></div><div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">===================<br>
Ye Luo, Ph.D.<br>Computational Science Division & Leadership Computing Facility<br>
Argonne National Laboratory</div></div></div></div></div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, May 28, 2020 at 2:34 PM Tobias Klöffel <<a href="mailto:tobias.kloeffel@fau.de">tobias.kloeffel@fau.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div>Dear all,</div>
<div><br>
</div>
<div>just to share my limited experience
with TSUBASA (please correct me if its wrong):</div>
<div><br>
</div>
<div>You can use it with any standard
C/C++/Fortran MPI/OpenMP code, the code runs completely on the
card. There is some work for back loading onto the host CPU and
also some work on using the card in a offload fashion, but I have
no clue about the status of these projects. If I remember
correctly those projects are C/C++ LLVM only.<br>
</div>
<div>Vectorizable code naturally performs
exceptionally good, however, unvectorizable code is only performed
via an ARM core and you will for sure find some routines that are
extremely slow and more or less irrelevant on standard CPUs.</div>
<div><br>
</div>
<div>NEC provides
BLAS/FFTW/Lapack/SCALapack, compiling CPMD was possible out of the
box. My biggest issue was indeed the support of OpenMP
parallelization in those libraries as well as support for nested
OpenMP in general.<br>
</div>
<div><br>
</div>
<div>However, I am looking forward to see
the code changes done by NEC, and of course, the performance.<br>
</div>
<div><br>
</div>
<div>Best,</div>
<div>Tobias</div>
<div><br>
</div>
<div>On 5/28/20 3:21 PM, IWATA NAOKI(岩田 直樹)
wrote:<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US">Dear Ye and Paolo,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US">Thank you very much for your quick response and
your interest in SX-Aurora TSUBASA.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US">I will answer your questions and will provide
the code to you later.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US">At this moment, please look at the following
site:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US"><a href="https://www.hpc.nec/index.en" target="_blank">https://www.hpc.nec/index.en</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US">Best Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US">Naoki Iwata.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:游ゴシック;color:rgb(31,73,125)" lang="EN-US"><u></u> <u></u></span></p>
<div style="border-color:currentcolor currentcolor currentcolor blue;border-style:none none none solid;border-width:medium medium medium 1.5pt;padding:0mm 0mm 0mm 4pt">
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0mm 0mm">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US"> Ye Luo <a href="mailto:xw111luoye@gmail.com" target="_blank"><xw111luoye@gmail.com></a>
<br>
<b>Sent:</b> Thursday, May 28, 2020 1:09 AM<br>
<b>To:</b> General discussion list for Quantum
ESPRESSO developers
<a href="mailto:developers@lists.quantum-espresso.org" target="_blank"><developers@lists.quantum-espresso.org></a><br>
<b>Cc:</b> IWATA NAOKI(</span><span style="font-size:11pt">岩田 直樹</span><span style="font-size:11pt;font-family:"Calibri",sans-serif" lang="EN-US">) <a href="mailto:n-iwata-da@nec.com" target="_blank"><n-iwata-da@nec.com></a><br>
<b>Subject:</b> ##freemail## Re: [QE-developers]
SX-Aurora TSUBASA<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hi Naoki,<u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Could you put
the code on gitlab or github for us to preview?<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Could you say a
bit more how did you "port"? Which part of QE was
ported? How did you port?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">NEC aurora is an
accelerator but it is not a GPU. How did you program
it?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">I was aware of FFT
libraries for SX-Aurora. Did you use SX-Aurora via
library?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">The code path to
SX in FFTXLIB is not maintained for a long time I
guess. Perhaps it is time to clean/refactor that part.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Regarding compiler
options, do you have a customized compiler? Or it is
gfortran or flang and you just need to just pass some
-D flags?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Best,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US">Ye<u></u><u></u></span></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">===================<br>
Ye Luo, Ph.D.<br>
Computational Science Division &
Leadership Computing Facility<br>
Argonne National Laboratory<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">On Wed, May 27,
2020 at 10:15 AM IWATA NAOKI(</span>岩田 直樹<span lang="EN-US">) <<a href="mailto:n-iwata-da@nec.com" target="_blank">n-iwata-da@nec.com</a>>
wrote:<u></u><u></u></span></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0mm 0mm 0mm 6pt;margin-left:4.8pt;margin-right:0mm">
<p class="MsoNormal"><span lang="EN-US">Dear developers,<br>
<br>
My name Naoki Iwata from NEC Corporation in Japan.<br>
We are developing and selling a newer
vector-supercomputer, SX-Aurora TSUBASA.<br>
<a href="https://www.nec.com/en/global/solutions/hpc/" target="_blank">https://www.nec.com/en/global/solutions/hpc/</a><br>
<br>
We have ported and optimized QUANTAM ESPRESSO v6.3 for
SX-Aurora TSUBASA.<br>
So we would like to ask you to implement our compile
options and source code changes
<br>
for the optimization for SX-Aurora TSUBASA into the
original QUANTUM ESPRESSO <br>
makefile and source code.<br>
I looked at the users' guide, I found the description
of SX-6 and SX-8. They are old system.<br>
So we also would like to update it to SX-Aurora
TSUBASA.<br>
What should we do for that?<br>
<br>
Best Regards,<br>
<br>
Naoki Iwata<br>
-----<br>
Naoki Iwata (<a href="mailto:n-iwata-da@nec.com" target="_blank">n-iwata-da@nec.com</a>)<br>
NEC AI Platform Division<br>
<br>
_______________________________________________<br>
developers mailing list<br>
<a href="mailto:developers@lists.quantum-espresso.org" target="_blank">developers@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/developers" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/developers</a><u></u><u></u></span></p>
</blockquote>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
developers mailing list
<a href="mailto:developers@lists.quantum-espresso.org" target="_blank">developers@lists.quantum-espresso.org</a>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/developers" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/developers</a>
</pre>
</blockquote>
<p><br>
</p>
<pre cols="72">--
M.Sc. Tobias Klöffel
=======================================================
HPC (High Performance Computing) group
Erlangen Regional Computing Center(RRZE)
Friedrich-Alexander-Universität Erlangen-Nürnberg
Martensstr. 1
91058 Erlangen
Room: 1.133
Phone: +49 (0) 9131 / 85 - 20101
=======================================================
E-mail: <a href="mailto:tobias.kloeffel@fau.de" target="_blank">tobias.kloeffel@fau.de</a></pre>
</div>
_______________________________________________<br>
developers mailing list<br>
<a href="mailto:developers@lists.quantum-espresso.org" target="_blank">developers@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/developers" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/developers</a><br>
</blockquote></div>