<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
Thank you for the reply,
<div><br>
</div>
<div>I use QE7.3. The qe7.0gpu module on NERSC is not working probably due to environment change. </div>
<div>My script compilation script is : </div>
<div><br>
</div>
<div>
<div>export LC_ALL=C</div>
<div>module load gpu</div>
<div>module load PrgEnv-nvidia</div>
<div>module load nvidia/23.9</div>
<div>module unload cray-libsci</div>
<div>module unload darshan</div>
<div>module load cudatoolkit/12.2</div>
<div>module load craype-accel-nvidia80</div>
<div>module load cray-python/3.11.5</div>
<div>module load cray-fftw</div>
<div>./configure --with-cuda=$CUDA_HOME --with-cuda-runtime=12.2 --with-cuda-cc=80 --with-cuda-mpi=yes</div>
<div><br>
</div>
<div>I manually made change to the make.inc as follow, and then make all: </div>
<div>
<div> MPIF90 = ftn</div>
<div>F90 = ftn</div>
<div> CC = cc</div>
<div> LD = ftn</div>
<div>BLAS_LIBS = ${CRAY_NVIDIA_PREFIX}/compilers/lib/libblas.so</div>
<div>LAPACK_LIBS = ${CRAY_NVIDIA_PREFIX}/compilers/lib/liblapack.so ${CRAY_NVIDIA_PREFIX}/compilers/lib/libblas.so</div>
<div>FFT_LIBS = $(FFTW_DIR)/libfftw3.so \</div>
<div> $(FFTW_DIR)/libfftw3_threads.so \</div>
<div> $(FFTW_DIR)/libfftw3_omp.so \</div>
<div> ${CUDALIB} -lstdc++</div>
</div>
<div><br>
</div>
<div>The code could run with no problem, but it gives different result from gpu compilation. </div>
<div><br>
</div>
<div>Best,</div>
<div>Shimin Zhang</div>
<div><br>
</div>
<div>University of Wisconsin, Madison </div>
<div>
<div><br>
<blockquote type="cite">
<div>On Jul 29, 2024, at 2:00 PM, Omar A. Ashour <ashour@berkeley.edu> wrote:</div>
<br class="Apple-interchange-newline">
<div>
<div>
<div dir="ltr">Hi Shimin,</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">You did not specify the QE version you are using, how it was compiled, and which libraries you linked to. If you’re using the default modules on NERSC’s Perlmutter (7.0-libxc5.2.2), they are linked to different libraries. I believe the CPU version
is compiled with GNU compilers and linked to Cray LibSci and FFTW3, whereas the GPU version is compiled with nvcc/PGI and linked to the NVIDIA implementations (cuBLAS, cuFFTW, etc.). In principle, they should give identical results that are close to numerical
precision, but there might be something going on here. </div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Your calculation is too expensive to try myself with my NERSC builds of QE 7.3. Sharing the information I suggested above may allow other people to help.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Omar A. Ashour</div>
<div dir="ltr">UC Berkeley</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">
<div dir="ltr">P.S. Please include your affiliation when posting to the mailing list.</div>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Jul 29, 2024 at 11:20:07 AM, Shimin ZHANG via users <<a href="mailto:users@lists.quantum-espresso.org">users@lists.quantum-espresso.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" type="cite">
<div>
<div></div>
<div style="line-break:after-white-space">Dear QE developers,
<div><br>
</div>
<div>I am running a PBE0 scf calculation on NERSC gpu. I found my gpu result are different from cpu result. I tested different cpu installation on different clusters and it gives <span style="">Bandgap=3.0945 eV and </span><span style="">!! total energy =
-41479.47524618 Ry . With gpu version I got </span><span style="">bandgap=2.9022 and </span><span style="">!! total energy = -41428.12547634 Ry. I got this result consistently by adjusting different cutoff energy and mixing_beta. </span></div>
<div><span style=""><br>
</span></div>
<div><font><span style="">Does anyone have similar experience on getting different result with gpu? </span></font></div>
<div><font><span style=""><br>
</span></font></div>
<div><font><span style="">Best,</span></font></div>
<div><font><span style="">Shimin</span></font></div>
<div><font><span style=""><br>
</span></font></div>
<div><font><span style="">Here is my input: </span></font></div>
<div><font><span style=""><br>
</span></font></div>
<div><font><span style=""><br>
</span></font></div>
<div>
<div>&control</div>
<div> calculation = 'scf'</div>
<div> wf_collect = .true.</div>
<div> verbosity = 'high'</div>
<div> pseudo_dir=“~/Pseudopotential/pseudo-ONCV-proj"</div>
<div> etot_conv_thr=1d-6,</div>
<div> forc_conv_thr=1d-5,</div>
<div> nstep=1000</div>
<div>/</div>
<div><br>
</div>
<div>&system</div>
<div> input_dft = "pbe0"</div>
<div> ibrav = 0</div>
<div> nat = 192</div>
<div> ntyp = 2</div>
<div> ecutwfc = 60</div>
<div> ecutfock = 120</div>
<div> nbnd = 1258</div>
<div>/</div>
<div><br>
</div>
<div>&electrons</div>
<div> conv_thr = 1e-09</div>
<div> electron_maxstep = 100</div>
<div> mixing_beta = 0.5</div>
<div>/</div>
<div>&ions</div>
<div>/</div>
<div>&cell</div>
<div>/</div>
<div>ATOMIC_SPECIES</div>
<div>Zn 0.0 Zn_ONCV_PBE-1.0.upf</div>
<div>O 0.0 O_ONCV_PBE-1.0.upf</div>
</div>
<div>
<div>K_POINTS gamma</div>
<div><br>
</div>
<div>CELL_PARAMETERS angstrom</div>
<div> 12.994354258 0.000000000 0.000000000</div>
<div> -6.497177129 11.253440893 0.000000000</div>
<div> 0.000000000 0.000000000 15.612038607</div>
</div>
<div>
<div>ATOMIC_POSITIONS crystal</div>
<div> O 0.333333250 0.916666750 0.793874404</div>
<div> O 0.583333250 0.666666750 0.127207737</div>
<div> O 0.666666750 0.583333250 0.293874404</div>
<div> O 0.583333250 0.416666750 0.793874404</div>
<div> O 0.666666750 0.333333250 0.960541070</div>
<div> O 0.583333250 0.416666750 0.460541070</div>
<div> O 0.666666750 0.333333250 0.627207737</div>
<div> O 0.583333250 0.416666750 0.127207737</div>
<div> O 0.666666750 0.333333250 0.293874404</div>
<div> O 0.583333250 0.166666750 0.793874404</div>
<div> O 0.666666750 0.083333250 0.960541070</div>
<div> O 0.583333250 0.166666750 0.460541070</div>
<div> O 0.666666750 0.083333250 0.627207737</div>
<div> O 0.583333250 0.166666750 0.127207737</div>
<div> O 0.666666750 0.583333250 0.627207737</div>
<div> O 0.666666750 0.083333250 0.293874404</div>
<div> O 0.416666750 0.833333250 0.960541070</div>
<div> O 0.333333250 0.916666750 0.460541070</div>
<div> O 0.416666750 0.833333250 0.627207737</div>
<div> O 0.333333250 0.916666750 0.127207737</div>
<div> O 0.416666750 0.833333250 0.293874404</div>
<div> O 0.333333250 0.666666750 0.793874404</div>
<div> O 0.416666750 0.583333250 0.960541070</div>
<div> O 0.333333250 0.666666750 0.460541070</div>
<div> O 0.416666750 0.583333250 0.627207737</div>
<div> O 0.333333250 0.666666750 0.127207737</div>
<div> O 0.416666750 0.583333250 0.293874404</div>
<div> O 0.333333250 0.416666750 0.793874404</div>
<div> O 0.416666750 0.333333250 0.960541070</div>
<div> O 0.916666750 0.833333250 0.960541070</div>
<div> O 0.333333250 0.416666750 0.460541070</div>
<div> O 0.583333250 0.666666750 0.460541070</div>
<div> O 0.583333250 0.666666750 0.793874404</div>
<div> O 0.833333250 0.916666750 0.460541070</div>
<div> O 0.916666750 0.833333250 0.627207737</div>
<div> O 0.833333250 0.916666750 0.127207737</div>
<div> O 0.916666750 0.833333250 0.293874404</div>
<div> O 0.833333250 0.666666750 0.793874404</div>
<div> O 0.916666750 0.583333250 0.960541070</div>
<div> O 0.833333250 0.666666750 0.460541070</div>
<div> O 0.916666750 0.583333250 0.627207737</div>
</div>
<div>
<div> O 0.833333250 0.666666750 0.127207737</div>
<div> O 0.916666750 0.583333250 0.293874404</div>
<div> O 0.833333250 0.416666750 0.793874404</div>
<div> O 0.916666750 0.333333250 0.960541070</div>
<div> O 0.833333250 0.416666750 0.460541070</div>
<div> O 0.666666750 0.583333250 0.960541070</div>
<div> O 0.916666750 0.333333250 0.627207737</div>
<div> O 0.916666750 0.333333250 0.293874404</div>
<div> O 0.833333250 0.166666750 0.793874404</div>
<div> O 0.916666750 0.083333250 0.960541070</div>
<div> O 0.833333250 0.166666750 0.460541070</div>
<div> O 0.916666750 0.083333250 0.627207737</div>
<div> O 0.833333250 0.166666750 0.127207737</div>
<div> O 0.916666750 0.083333250 0.293874404</div>
<div> O 0.583333250 0.916666750 0.793874404</div>
<div> O 0.666666750 0.833333250 0.960541070</div>
<div> O 0.583333250 0.916666750 0.460541070</div>
<div> O 0.666666750 0.833333250 0.627207737</div>
<div> O 0.583333250 0.916666750 0.127207737</div>
<div> O 0.666666750 0.833333250 0.293874404</div>
<div> O 0.833333250 0.416666750 0.127207737</div>
<div> O 0.416666750 0.333333250 0.627207737</div>
<div> O 0.833333250 0.916666750 0.793874404</div>
</div>
<div>
<div> O 0.083333250 0.666666750 0.460541070</div>
<div> O 0.166666750 0.083333250 0.960541070</div>
<div> O 0.083333250 0.166666750 0.793874404</div>
<div> O 0.083333250 0.916666750 0.793874404</div>
<div> O 0.166666750 0.833333250 0.960541070</div>
<div> O 0.166666750 0.333333250 0.293874404</div>
<div> O 0.083333250 0.916666750 0.460541070</div>
<div> O 0.083333250 0.416666750 0.127207737</div>
<div> O 0.166666750 0.833333250 0.627207737</div>
<div> O 0.083333250 0.916666750 0.127207737</div>
<div> O 0.166666750 0.333333250 0.627207737</div>
<div> O 0.166666750 0.833333250 0.293874404</div>
<div> O 0.083333250 0.416666750 0.460541070</div>
<div> O 0.083333250 0.666666750 0.793874404</div>
<div> O 0.166666750 0.583333250 0.960541070</div>
<div> O 0.166666750 0.333333250 0.960541070</div>
<div> O 0.083333250 0.416666750 0.793874404</div>
<div> O 0.333333250 0.416666750 0.127207737</div>
</div>
<div>
<div> O 0.166666750 0.583333250 0.627207737</div>
<div> O 0.166666750 0.583333250 0.293874404</div>
<div> O 0.416666750 0.083333250 0.293874404</div>
<div> O 0.333333250 0.166666750 0.127207737</div>
<div> O 0.083333250 0.666666750 0.127207737</div>
<div> O 0.416666750 0.083333250 0.960541070</div>
<div> O 0.083333250 0.166666750 0.460541070</div>
<div> O 0.416666750 0.333333250 0.293874404</div>
<div> O 0.416666750 0.083333250 0.627207737</div>
<div> O 0.333333250 0.166666750 0.793874404</div>
<div> O 0.166666750 0.083333250 0.627207737</div>
<div> O 0.083333250 0.166666750 0.127207737</div>
<div> O 0.333333250 0.166666750 0.460541070</div>
<div> O 0.166666750 0.083333250 0.293874404</div>
<div> Zn 0.916666750 0.083333250 0.833358930</div>
<div> Zn 0.916666750 0.833333250 0.500025596</div>
<div> Zn 0.166666750 0.333333250 0.833358930</div>
<div> Zn 0.833333250 0.166666750 0.333358930</div>
<div> Zn 0.916666750 0.083333250 0.500025596</div>
<div> Zn 0.083333250 0.416666750 0.666692263</div>
<div> Zn 0.833333250 0.916666750 0.333358930</div>
<div> Zn 0.166666750 0.083333250 0.833358930</div>
<div> Zn 0.083333250 0.166666750 0.000025596</div>
<div> Zn 0.916666750 0.833333250 0.833358930</div>
<div> Zn 0.166666750 0.583333250 0.166692263</div>
<div> Zn 0.583333250 0.916666750 0.666692263</div>
<div> Zn 0.666666750 0.833333250 0.833358930</div>
<div> Zn 0.083333250 0.666666750 0.000025596</div>
<div> Zn 0.833333250 0.916666750 0.666692263</div>
<div> Zn 0.833333250 0.166666750 0.000025596</div>
<div> Zn 0.916666750 0.083333250 0.166692263</div>
<div> Zn 0.166666750 0.083333250 0.500025596</div>
<div> Zn 0.833333250 0.666666750 0.333358930</div>
<div> Zn 0.916666750 0.583333250 0.500025596</div>
<div> Zn 0.083333250 0.166666750 0.666692263</div>
<div> Zn 0.916666750 0.583333250 0.833358930</div>
<div> Zn 0.833333250 0.666666750 0.000025596</div>
<div> Zn 0.916666750 0.583333250 0.166692263</div>
<div> Zn 0.833333250 0.666666750 0.666692263</div>
</div>
<div>
<div> Zn 0.166666750 0.333333250 0.166692263</div>
<div> Zn 0.833333250 0.416666750 0.666692263</div>
<div> Zn 0.583333250 0.916666750 0.333358930</div>
<div> Zn 0.083333250 0.416666750 0.000025596</div>
<div> Zn 0.833333250 0.416666750 0.333358930</div>
<div> Zn 0.083333250 0.166666750 0.333358930</div>
<div> Zn 0.916666750 0.333333250 0.500025596</div>
<div> Zn 0.916666750 0.833333250 0.166692263</div>
<div> Zn 0.833333250 0.916666750 0.000025596</div>
<div> Zn 0.166666750 0.333333250 0.500025596</div>
<div> Zn 0.833333250 0.416666750 0.000025596</div>
<div> Zn 0.916666750 0.333333250 0.166692263</div>
<div> Zn 0.083333250 0.416666750 0.333358930</div>
<div> Zn 0.833333250 0.166666750 0.666692263</div>
<div> Zn 0.916666750 0.333333250 0.833358930</div>
<div> Zn 0.416666750 0.333333250 0.500025596</div>
<div> Zn 0.166666750 0.583333250 0.500025596</div>
<div> Zn 0.666666750 0.083333250 0.166692263</div>
<div> Zn 0.416666750 0.083333250 0.166692263</div>
<div> Zn 0.333333250 0.916666750 0.666692263</div>
<div> Zn 0.416666750 0.833333250 0.833358930</div>
<div> Zn 0.333333250 0.166666750 0.000025596</div>
<div> Zn 0.333333250 0.916666750 0.333358930</div>
<div> Zn 0.416666750 0.833333250 0.500025596</div>
<div> Zn 0.416666750 0.083333250 0.500025596</div>
<div> Zn 0.333333250 0.916666750 0.000025596</div>
<div> Zn 0.416666750 0.833333250 0.166692263</div>
<div> Zn 0.333333250 0.166666750 0.333358930</div>
<div> Zn 0.583333250 0.166666750 0.000025596</div>
<div> Zn 0.333333250 0.666666750 0.666692263</div>
<div> Zn 0.416666750 0.083333250 0.833358930</div>
<div> Zn 0.333333250 0.666666750 0.333358930</div>
<div> Zn 0.416666750 0.583333250 0.500025596</div>
<div> Zn 0.333333250 0.166666750 0.666692263</div>
<div> Zn 0.333333250 0.666666750 0.000025596</div>
<div> Zn 0.416666750 0.583333250 0.166692263</div>
<div> Zn 0.416666750 0.333333250 0.166692263</div>
<div> Zn 0.333333250 0.416666750 0.666692263</div>
<div> Zn 0.416666750 0.333333250 0.833358930</div>
</div>
<div>
<div> Zn 0.333333250 0.416666750 0.000025596</div>
<div> Zn 0.333333250 0.416666750 0.333358930</div>
<div> Zn 0.416666750 0.583333250 0.833358930</div>
<div> Zn 0.666666750 0.833333250 0.500025596</div>
<div> Zn 0.083333250 0.916666750 0.666692263</div>
<div> Zn 0.583333250 0.166666750 0.333358930</div>
<div> Zn 0.583333250 0.916666750 0.000025596</div>
<div> Zn 0.666666750 0.833333250 0.166692263</div>
<div> Zn 0.083333250 0.666666750 0.333358930</div>
<div> Zn 0.583333250 0.666666750 0.666692263</div>
<div> Zn 0.666666750 0.583333250 0.833358930</div>
<div> Zn 0.166666750 0.583333250 0.833358930</div>
<div> Zn 0.583333250 0.666666750 0.333358930</div>
<div> Zn 0.666666750 0.583333250 0.500025596</div>
<div> Zn 0.083333250 0.666666750 0.666692263</div>
<div> Zn 0.583333250 0.666666750 0.000025596</div>
<div> Zn 0.666666750 0.583333250 0.166692263</div>
<div> Zn 0.666666750 0.083333250 0.500025596</div>
<div> Zn 0.166666750 0.833333250 0.166692263</div>
<div> Zn 0.666666750 0.333333250 0.833358930</div>
<div> Zn 0.083333250 0.916666750 0.000025596</div>
<div> Zn 0.583333250 0.416666750 0.333358930</div>
<div> Zn 0.666666750 0.333333250 0.500025596</div>
<div> Zn 0.166666750 0.833333250 0.500025596</div>
<div> Zn 0.583333250 0.416666750 0.000025596</div>
<div> Zn 0.666666750 0.333333250 0.166692263</div>
<div> Zn 0.083333250 0.916666750 0.333358930</div>
<div> Zn 0.583333250 0.166666750 0.666692263</div>
<div> Zn 0.666666750 0.083333250 0.833358930</div>
<div> Zn 0.166666750 0.833333250 0.833358930</div>
<div> Zn 0.583333250 0.416666750 0.666692263</div>
<div> Zn 0.166666750 0.083333250 0.166692263</div>
</div>
<div><br>
</div>
</div>
</div>
<div>
<div>_______________________________________________<br>
The Quantum ESPRESSO community stands by the Ukrainian<br>
people and expresses its concerns about the devastating<br>
effects that the Russian military offensive has on their<br>
country and on the free and peaceful scientific, cultural,<br>
and economic cooperation amongst peoples<br>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="https://urldefense.com/v3/__http://www.max-centre.eu__;!!Mak6IKo!JE0FxXeLf_JVmtda3e1vdd3gzzVhRoSNYwoxcpQBJo0_e0vyMEqBRTC5poR0zJSGUMysQLGZVZz_RCYYnPeuiTc$">www.max-centre.eu</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org">users@lists.quantum-espresso.org</a><br>
<a href="https://urldefense.com/v3/__https://lists.quantum-espresso.org/mailman/listinfo/users__;!!Mak6IKo!JE0FxXeLf_JVmtda3e1vdd3gzzVhRoSNYwoxcpQBJo0_e0vyMEqBRTC5poR0zJSGUMysQLGZVZz_RCYYbSI7Cco$">https://lists.quantum-espresso.org/mailman/listinfo/users</a>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>