<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: style=" font-size:="" 12pt;="" color:="" #000000"="">Dear Pietro,<br>Thanks a lot for your reply.<br>I will try out your suggestions and check.<br><br>With regards,<br>Niharika<br><br><hr id="zwchr" data-marker=""><div data-marker=""><b>From: </b>pdelugas@sissa.it<br><b>To: </b>users@lists.quantum-espresso.org<br><b>Sent: </b>Tuesday, October 15, 2024 2:02:33 PM<br><b>Subject: </b>Re: [QE-users] Large time lag post software upgradation in HPC system<br></div><br><div data-marker=""><style style="display:none">/*<![CDATA[*/P {
margin-top: 0;
margin-bottom: 0;
}
/*]]>*/</style>
<div style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
Hello</div>
<div style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
It is strange because qe v7.3 is way faster than 6.7, especially on GPUs. It has to do with some fine-tuning in using the cluster.</div>
<div style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
You should ask help to the system managers of your cluster.</div>
<div style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
Just trying to guess:</div>
<ol start="1"><li style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
<div>The problem might be hyperthreading, so make sure that OMP_NUM_THREADS is set to 1.</div>
</li><li style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
<div>try to see in the GPU MPI aware communications are working compile with --with-cuda-mpi=no</div>
</li></ol>
<div style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
<br>
</div>
<div style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
hope it helps</div>
<div style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
best regards</div>
<div style="font-family:'aptos' , 'aptos_embeddedfont' , 'aptos_msfontservice' , 'calibri' , 'helvetica' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )">
Pietro</div>
<hr style="display:inline-block;width:98%">
<div dir="ltr"><span style="font-family:'calibri' , sans-serif;font-size:11pt;color:rgb( 0 , 0 , 0 )"><b>From:</b> users <users-bounces@lists.quantum-espresso.org> on behalf of Niharika Joshi <nh.joshi@ncl.res.in><br>
<b>Sent:</b> Tuesday, October 15, 2024 09:34<br>
<b>To:</b> Quantum ESPRESSO users Forum <users@lists.quantum-espresso.org><br>
<b>Subject:</b> [QE-users] Large time lag post software upgradation in HPC system</span>
<div> </div>
</div>
<div style="font-family:'times new roman' , 'new york' , serif">Dear QE users,</div>
<div style="font-family:'times new roman' , 'new york' , serif">I am using a HPC resource for more than a year with QE(6.7Max GPU) without any issue. My present research problem focuses on studying methane and carbon dioxide adsorption on spinel surfaces. The
system is large with more than 380 atoms and ~3500 electrons. Normally, 2-3 ionic cycles (with 60-70 iterations) gets complete within a day. However, recently there has been some software upgradation in the computing system after which I have observed a huge
time lag in my calculations. Currently, only few iterations are performed in 24 hours.</div>
<div style="font-family:'times new roman' , 'new york' , serif"><br>
</div>
<div style="font-family:'times new roman' , 'new york' , serif">Please find below two tables listing the details of hardware specifications and upgradation information of software in the computing system. </div>
<span style="font-family:'arial' , 'helvetica' , sans-serif"><br>
</span>
<table style="width:62.8258%;border-collapse:collapse;border-spacing:0px"><tbody><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
<b>Component</b> </div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
<b>Specification</b></div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
CPU</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
AMD EPYC 7742 64C 2.25GHz</div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
CPU core</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
128 cores (Dual socket each with 64 cores); 256 cores with hyper-threading</div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
L3 cache</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
256 Mb</div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
RAM</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
1 TB</div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
GPU</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
NVIDIA A100-SXM4</div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
GPU Memory</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
40 Gb</div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
Total no. of GPU per node</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
8</div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
Storage</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
10.5 PiB PFS based storage</div>
</td></tr><tr><td style="text-align:center;width:19.7936%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
Networking</div>
</td><td style="text-align:center;width:53.0322%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
Mellonex ConnectX-6 VPI (infiniband HDR)</div>
</td></tr></tbody></table>
<span style="font-family:'arial' , 'helvetica' , sans-serif"><br>
</span>
<div style="font-family:'arial' , 'helvetica' , sans-serif"><br>
</div>
<table style="width:49.56%;border-collapse:collapse;border-spacing:0px"><tbody><tr><td style="text-align:center;width:12.0685%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
<b>Software</b></div>
</td><td style="text-align:center;width:37.4915%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
<b>Specification of upgradation</b></div>
</td></tr><tr><td style="text-align:center;width:12.0685%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
OS</div>
</td><td style="text-align:center;width:37.4915%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
from Ubuntu 20.04.02 (DGX OS 5.0.5) to Ubuntu 22.04.04 (DGX OS 6.3.0)</div>
</td></tr><tr><td style="text-align:center;width:12.0685%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
Kernel</div>
</td><td style="text-align:center;width:37.4915%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
from 5.4.0-80-generic to 5.15.0-1062-nvidia</div>
</td></tr><tr><td style="text-align:center;width:12.0685%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
CUDA</div>
</td><td style="text-align:center;width:37.4915%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
10.1 to 12.4 (below versions are also available)</div>
</td></tr><tr><td style="text-align:center;width:12.0685%">
<div style="text-align:center;font-family:'times new roman' , 'new york' , serif">
NVIDIA Driver version</div>
</td><td style="text-align:center;width:37.4915%">
<div style="text-align:center"><span style="font-family:'times new roman' , 'new york' , serif">450.142.00 to 550.90.07</span><span style="font-family:'arial' , 'helvetica' , sans-serif"><br>
</span></div>
</td></tr></tbody></table>
<div style="font-family:'arial' , 'helvetica' , sans-serif"><br>
</div>
<div style="font-family:'times new roman' , 'new york' , serif">Post software upgradation, QE-7.3 was installed in the following manner:</div>
<pre></pre><div style="text-indent:0px;white-space:pre-wrap;color:rgb( 0 , 0 , 0 )"><span style="font-family:'times new roman' , 'new york' , serif;font-size:12pt"><b>Step 1</b> : Source up the HPC-SDK environment:
source /opt/hpc-sdk-23.9/env.sh
<b>Step 2.</b> Set up the environment:
./configure --prefix=installation-location --with-cuda=$CUDA_ROOT --with-cuda-runtime=12.2 --with-cuda-cc=80 --enable-openmp --with-scalapack=no --with-cuda-mpi=yes
<b>Step 3.</b> Compile the source code:
make all -j8
<b>Step 4</b>. Install the compiled binaries:
make instal</span><span style="font-family:'times new roman' , 'new york' , serif">l </span><br><br><span style="font-family:'times new roman' , 'new york' , serif">Kindly, suggest some solution to this problem. Any advice/suggestion at this point would really be very helpful to me.</span><br><br><span style="font-family:'times new roman' , 'new york' , serif">With best regards,</span><br><span style="font-family:'times new roman' , 'new york' , serif">Niharika Joshi,</span><br><span style="font-family:'times new roman' , 'new york' , serif">National Post Doctoral Fellow,</span><br><span style="font-family:'times new roman' , 'new york' , serif">CSIR National Chemical Laboratory, Pune,</span><br><span style="font-family:'times new roman' , 'new york' , serif">Maharashtra-411008, India. </span></div>
<br>
<br>
<br>
<br>_______________________________________________<br>The Quantum ESPRESSO community stands by the Ukrainian<br>people and expresses its concerns about the devastating<br>effects that the Russian military offensive has on their<br>country and on the free and peaceful scientific, cultural,<br>and economic cooperation amongst peoples<br>_______________________________________________<br>Quantum ESPRESSO is supported by MaX (www.max-centre.eu)<br>users mailing list users@lists.quantum-espresso.org<br>https://lists.quantum-espresso.org/mailman/listinfo/users<br></div></div></body></html>