<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</p>
<div class="field field-name-body field-type-text-with-summary
field-label-hidden" style="color: rgb(96, 96, 96); font-family:
Arial, 宋体, Tahoma, Helvetica, sans-serif; font-size: 13.008px;
font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: normal; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);">
<div class="field-items">
<div class="field-item even" property="content:encoded">
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">Dear QE users,</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">I am referring to the guide on Explicit
offload QE to Xeon Phi KNC (7120P) here,</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);"><a
href="https://software.intel.com/en-us/articles/explicit-offload-for-quantum-espresso"
style="margin: 0px; padding: 0px; color: rgb(8, 109, 182);
text-decoration: none; font-family: Arial, 宋体, Tahoma,
Helvetica, sans-serif; font-weight: normal;">https://software.intel.com/en-us/articles/explicit-offload-for-quantum-espresso</a></p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">I tried to follow the above steps but I
failed to run pw.x (QE v5.3.0) on 2 Xeon Phi 7120P using the
mpirun.sh script. The error reads:</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);"> allocating buffers 2048
2048 1024<br>
on device 0<br>
threshold 20000000000.0000 <br>
allocating buffers 2048 2048 1024<br>
on device 0<br>
threshold 20000000000.0000 <br>
offload error: cannot create buffer on device 0 (error code
14)<br>
offload error: cannot create buffer on device 0 (error code
14)</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">This is how I run the script,</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">[qeuser@node09 ~]$ ~/mpirun/mpirun.sh -p 1
-w ~/libxphi/xphilibwrapper.sh -x
~/QE530-KNC-OL/espresso-5.3.0/bin/pw.x -i
~/rolly/AUSURF112/ausurf.in</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">I have already scp all the lib and bin
files to each Xeon Phi 7120P and I have also compiled the
libxphi lib. This is how the libxphi directory reads,</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">[qeuser@node09 libxphi]$ ls<br>
build-library.sh libmkl_proxy.so LICENSE README.md
xphilibmod.mod xphilib.o xphilib_proxy.o<br>
clean.sh libxphi.so mkl_proxy.c xphilib.f90
xphilibmod.modmic xphilib_proxy.f90 xphilibwrapper.sh</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">I suppose this is okay.</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">However, I found it interesting that I can
run a single instance on mic0 but it is very slow. This is
how I did it,</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">[qeuser@node09 ~]$ export
LD_LIBRARY_PATH=/home/qeuser/libxphi/:$LD_LIBRARY_PATH</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">[qeuser@node09 ~]$
LD_PRELOAD="/home/qeuser/libxphi/libxphi.so" /home/qeuser/QE530-KNC-OL/espresso-5.3.0/bin/pw.x
< /home/qeuser/rolly/AUSURF112/ausurf.in</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">Error messages were also produced,</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);"> allocating buffers 2048
2048 1024<br>
on device 0<br>
threshold 20000000000.0000 <br>
ERROR: ld.so: object '/home/qeuser/libxphi/libxphi.so' from
LD_PRELOAD cannot be preloaded (cannot open shared object
file): ignored.<br>
ERROR: ld.so: object '/home/qeuser/libxphi/libxphi.so' from
LD_PRELOAD cannot be preloaded (cannot open shared object
file): ignored.<br>
buffer allocation 4.02019500732422 s</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">On the host I can see one copy of pw.x is
running, and on mic0 I can see that offload_main and
coi_daemon are running by the micuser. But it is very slow.</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">So, is this offload error: cannot create
buffer on device 0 (error code 14) related to the mpirun.sh
script and the libxphi.so were not preloaded even it is
present???</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">I am running CentOS 7.1 + Intel MPSS 3.8.1
+ Intel psxe 2017 update 1, and I have already made a
symbolic link of the psxevars.sh to /etc/profile.d and I can
use mpirun to pw.x on the host, but not offload to mic0 and
mic1</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">Are these compatibility issues because the
libxphi and mpirun.sh were written 2 years ago? How can
these be fixed?</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">Thank you,</p>
<p style="word-wrap: break-word; margin: 0px 0px 1.5em;
padding: 0px; font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 12px; font-weight: normal; color:
rgb(83, 87, 94);">Rolly</p>
</div>
</div>
</div>
<section class="field field-name-field-thread-topic
field-type-list-text field-label-above" style="display: block;
color: rgb(96, 96, 96); font-family: Arial, 宋体, Tahoma, Helvetica,
sans-serif; font-size: 13.008px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: normal; letter-spacing: normal; orphans: 2;
text-align: start; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255, 255,
255);">-- </section>
<pre class="moz-signature" cols="72">PhD. Research Fellow,
Dept. of Physics & Materials Science,
City University of Hong Kong
Tel: +852 3442 4000
Fax: +852 3442 0538</pre>
</body>
</html>