<div dir="ltr"><div>Dear Pietro,</div><div><br></div><div>Thanks for sharing the link. I got the error of missing v0.3.1.tar.gz file in the archive folder, so I downloaded it manually. Everything went well and I could run jobs. However, there are a few things that are unclear to me.</div><div><br></div><div>config options:<br></div><div><br></div><div>./configure FC=pgf90 F90=pgf90 CC=pgcc --with-cuda=yes --enable-cuda-env-check=no --with-cuda-runtime=11.0 --with-cuda-cc=70 --enable-openmp --with-scalapack=no</div><div><br></div><div>The make.sys file is changed based on the install instructions on this page: <a href="https://gitlab.com/QEF/q-e-gpu/-/wikis/home">https://gitlab.com/QEF/q-e-gpu/-/wikis/home</a></div><div><br></div><div>
DFLAGS = -D__CUDA -D__DFTI -D__MPI
</div><div>DFLAGS = -D__CUDA -D__DFTI -D__MPI__GPU_MPI</div><div><br></div><div>Since two cards are installed on the mainboard, I ran the jobs in this form:</div><div><br></div><div>mpirun -np 2 pw.x -input <a href="http://file.in">file.in</a> | tee file.out</div><div><br></div><div>The hardware
<span class="gmail-kgnlhe" tabindex="0" dir="ltr">specifications in the output file:</span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><br></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"> GPU used by master process:<br><br> Device Number: 0<br> Device name: TITAN V<br> Compute capability : 70<br> Ratio of single to double precision performance : 2<br> Memory Clock Rate (KHz): 850000<br> Memory Bus Width (bits): 3072<br> Peak Memory Bandwidth (GB/s): 652.80</span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><br></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr">I just wonder why the second card (device # 1) is not printed in the output.</span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><br></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr">Since
-D__DFTI is printed in the make.sys file, the MKL and FFTW of Intel Parallel Studio is utilized. Is this an appropriate config in order to get the best performance?<br></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><br></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr">Is it possible to compile the code for multiple graphic cards with different cuda-cc ? <br></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><br></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr">
<span class="gmail-q-box gmail-qu-userSelect--text" style="box-sizing:border-box"><span style="font-weight:normal;font-style:normal;background:rgba(0,0,0,0) none repeat scroll 0% 0%">I’m
sorry for asking so many questions. I appreciate your time in
responding to whatever you can, when you are able to find the time.</span></span></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><span class="gmail-q-box gmail-qu-userSelect--text" style="box-sizing:border-box"><span style="font-weight:normal;font-style:normal;background:rgba(0,0,0,0) none repeat scroll 0% 0%"><br></span></span></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><span class="gmail-q-box gmail-qu-userSelect--text" style="box-sizing:border-box"><span style="font-weight:normal;font-style:normal;background:rgba(0,0,0,0) none repeat scroll 0% 0%">Best,</span></span></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><span class="gmail-q-box gmail-qu-userSelect--text" style="box-sizing:border-box"><span style="font-weight:normal;font-style:normal;background:rgba(0,0,0,0) none repeat scroll 0% 0%"><br></span></span></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><span class="gmail-q-box gmail-qu-userSelect--text" style="box-sizing:border-box"><span style="font-weight:normal;font-style:normal;background:rgba(0,0,0,0) none repeat scroll 0% 0%">Mohammad</span></span></span></div><div><span class="gmail-kgnlhe" tabindex="0" dir="ltr"><span class="gmail-q-box gmail-qu-userSelect--text" style="box-sizing:border-box"><span style="font-weight:normal;font-style:normal;background:rgba(0,0,0,0) none repeat scroll 0% 0%">ShirazU<br></span></span>
</span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Dec 20, 2020 at 9:08 PM Pietro Bonfa' <<a href="mailto:pietro.bonfa@unipr.it">pietro.bonfa@unipr.it</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear Mohammad,<br>
<br>
for some reason you are having trouble accessing gitlab. I uploaded a <br>
package that includes all dependencies and can be compiled without <br>
network access. You can find it here:<br>
<br>
<a href="https://univpr-my.sharepoint.com/:u:/g/personal/pietro_bonfa_unipr_it/EV-nHENjf1lFkat0RvJypFIBap2o92v9BzG75po06z48WA?e=uiDjDD" rel="noreferrer" target="_blank">https://univpr-my.sharepoint.com/:u:/g/personal/pietro_bonfa_unipr_it/EV-nHENjf1lFkat0RvJypFIBap2o92v9BzG75po06z48WA?e=uiDjDD</a><br>
<br>
Best wishes,<br>
Pietro<br>
<br>
--<br>
Pietro Bonfà,<br>
University of Parma<br>
<br>
On 12/19/20 7:27 AM, Mohammad Moaddeli wrote:<br>
> Dear Louis and Pietro,<br>
> <br>
> *With the config options as following:*<br>
> *<br>
> *<br>
> *./configure FC=pgf90 F90=pgf90 CC=pgcc <br>
> --with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.0 <br>
> --with-cuda-runtime=11.0 --with-cuda-cc=70 --enable-openmp <br>
> --with-scalapack=no*<br>
> *<br>
> *<br>
> *results is:*<br>
> <br>
> checking build system type... x86_64-pc-linux-gnu<br>
> checking ARCH... x86_64<br>
> checking setting AR... ... ar<br>
> checking setting ARFLAGS... ... ruv<br>
> checking whether the Fortran compiler works... yes<br>
> checking for Fortran compiler default output file name... a.out<br>
> checking for suffix of executables...<br>
> checking whether we are cross compiling... no<br>
> checking for suffix of object files... o<br>
> checking whether we are using the GNU Fortran compiler... no<br>
> checking whether pgf90 accepts -g... yes<br>
> configure: WARNING: F90 value is set to be consistent with value of MPIF90<br>
> checking for mpiifort... no<br>
> checking for mpif90... mpif90<br>
> checking whether we are using the GNU Fortran compiler... no<br>
> checking whether mpif90 accepts -g... yes<br>
> checking version of mpif90... nvfortran 20.11-0<br>
> checking for Fortran flag to compile .f90 files... none<br>
> setting F90... nvfortran<br>
> setting MPIF90... mpif90<br>
> checking whether we are using the GNU C compiler... yes<br>
> checking whether pgcc accepts -g... yes<br>
> checking for pgcc option to accept ISO C89... none needed<br>
> setting CC... pgcc<br>
> setting CFLAGS... -fast -Mpreprocess<br>
> using F90... nvfortran<br>
> setting FFLAGS... -O1<br>
> setting F90FLAGS... $(FFLAGS)<br>
> setting FFLAGS_NOOPT... -O0<br>
> setting CPP... cpp<br>
> setting CPPFLAGS... -P -traditional -Uvector<br>
> setting LD... mpif90<br>
> setting LDFLAGS...<br>
> checking for Fortran flag to compile .f90 files... (cached) none<br>
> checking whether Fortran compiler accepts -Mcuda=cuda11.0... yes<br>
> checking for nvcc... <br>
> /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvcc<br>
> checking whether nvcc works... yes<br>
> checking for cuInit in -lcuda... no<br>
> configure: error: in `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7':<br>
> configure: error: Couldn't find libcuda<br>
> See `config.log' for more details<br>
> <br>
> *Although the option --enable-cuda-env-check=no resulted in the <br>
> configuration finished:*<br>
> *<br>
> *<br>
> *./configure FC=pgf90 F90=pgf90 CC=pgcc --with-cuda=yes <br>
> --enable-cuda-env-check=no --with-cuda-runtime=11.0 --with-cuda-cc=70 <br>
> --enable-openmp --with-scalapack=no*<br>
> <br>
> checking build system type... x86_64-pc-linux-gnu<br>
> checking ARCH... x86_64<br>
> checking setting AR... ... ar<br>
> checking setting ARFLAGS... ... ruv<br>
> checking whether the Fortran compiler works... yes<br>
> checking for Fortran compiler default output file name... a.out<br>
> checking for suffix of executables...<br>
> checking whether we are cross compiling... no<br>
> checking for suffix of object files... o<br>
> checking whether we are using the GNU Fortran compiler... no<br>
> checking whether pgf90 accepts -g... yes<br>
> configure: WARNING: F90 value is set to be consistent with value of MPIF90<br>
> checking for mpiifort... no<br>
> checking for mpif90... mpif90<br>
> checking whether we are using the GNU Fortran compiler... no<br>
> checking whether mpif90 accepts -g... yes<br>
> checking version of mpif90... nvfortran 20.11-0<br>
> checking for Fortran flag to compile .f90 files... none<br>
> setting F90... nvfortran<br>
> setting MPIF90... mpif90<br>
> checking whether we are using the GNU C compiler... yes<br>
> checking whether pgcc accepts -g... yes<br>
> checking for pgcc option to accept ISO C89... none needed<br>
> setting CC... pgcc<br>
> setting CFLAGS... -fast -Mpreprocess<br>
> using F90... nvfortran<br>
> setting FFLAGS... -O1<br>
> setting F90FLAGS... $(FFLAGS)<br>
> setting FFLAGS_NOOPT... -O0<br>
> setting CPP... cpp<br>
> setting CPPFLAGS... -P -traditional -Uvector<br>
> setting LD... mpif90<br>
> setting LDFLAGS...<br>
> checking for Fortran flag to compile .f90 files... (cached) none<br>
> checking whether Fortran compiler accepts -Mcuda=cuda11.0... yes<br>
> checking for /usr/local/cuda/... no<br>
> checking for /usr/local/cuda/include... no<br>
> checking for /usr/local/cuda/lib64... no<br>
> checking for nvcc... <br>
> /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvcc<br>
> checking whether nvcc works... yes<br>
> checking for cusolverDnZhegvdx_bufferSize in -lcusolver... no<br>
> configure: WARNING: Using legacy custom solver.<br>
> checking whether make sets $(MAKE)... yes<br>
> checking whether Fortran files must be preprocessed... yes<br>
> checking whether we are using the GNU Fortran 77 compiler... no<br>
> checking whether nvfortran accepts -g... yes<br>
> checking for library containing dgemm... no<br>
> MKL not found<br>
> in /opt/intel/mkl/lib/intel64: checking for library containing dgemm... <br>
> -lmkl_intel_lp64<br>
> setting BLAS_LIBS... -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 <br>
> -lmkl_intel_thread -lmkl_core<br>
> checking FFT...<br>
> checking MASS...<br>
> checking for library containing mpi_init... none required<br>
> checking ELPA...<br>
> checking BEEF... -lbeef<br>
> setting BEEF_LIBS... $(TOPDIR)/LIBBEEF/libbeef.a<br>
> checking for ranlib... ranlib<br>
> checking for wget... wget -O<br>
> setting WGET... wget -O<br>
> setting DFLAGS... -D__CUDA -D__DFTI -D__MPI<br>
> setting IFLAGS... -I$(TOPDIR)/include -I$(TOPDIR)/FoX/finclude <br>
> -I/opt/intel/mkl/include<br>
> configure: creating ./config.status<br>
> config.status: creating install/make_lapack.inc<br>
> config.status: creating include/configure.h<br>
> config.status: creating make.inc<br>
> config.status: creating configure.msg<br>
> config.status: creating install/make_wannier90.inc<br>
> config.status: creating include/qe_cdefs.h<br>
> --------------------------------------------------------------------<br>
> ESPRESSO can take advantage of several optimized numerical libraries<br>
> (essl, fftw, mkl...). This configure script attempts to find them,<br>
> but may fail if they have been installed in non-standard locations.<br>
> If a required library is not found, the local copy will be compiled.<br>
> <br>
> The following libraries have been found:<br>
> BLAS_LIBS=-L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64 <br>
> -lmkl_intel_thread -lmkl_core<br>
> LAPACK_LIBS=<br>
> FFT_LIBS=<br>
> <br>
> Please check if this is what you expect.<br>
> <br>
> If any libraries are missing, you may specify a list of directories<br>
> to search and retry, as follows:<br>
> ./configure LIBDIRS="list of directories, separated by spaces"<br>
> <br>
> Parallel environment detected successfully.\<br>
> Configured for compilation of parallel executables.<br>
> <br>
> For more info, read the ESPRESSO User's Guide (Doc/users-guide.tex).<br>
> --------------------------------------------------------------------<br>
> configure: success<br>
> <br>
> *make pw results in:*<br>
> <br>
> ....<br>
> .....<br>
> ....<br>
> cpp -P -traditional -Uvector -D__CUDA -D__DFTI -D__MPI <br>
> -I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//include <br>
> -I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//FoX/finclude <br>
> -I/opt/intel/mkl/include thread_util.f90 -o thread_util_tmp.f90 ; \<br>
> mpif90 -O1 -D__CUDA -D__DFTI -D__MPI -Mcuda=cc70,cuda11.0 <br>
> -I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//external/devxlib/src <br>
> -I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//external/devxlib/include <br>
> -I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//EIGENSOLVER_GPU/lib_eigsolve <br>
> -I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//external/devxlib/src <br>
> -I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//include <br>
> -I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//FoX/finclude <br>
> -I/opt/intel/mkl/include -I. -c thread_util_tmp.f90 -o thread_util.o<br>
> ar ruv libutil.a clocks_handler.o device_helper.o divide.o data_buffer.o <br>
> error_handler.o export_gstart_2_solvers.o find_free_unit.o <br>
> fletcher32_mod.o mem_counter.o mp.o mp_base.o mp_base_gpu.o <br>
> mp_bands_util.o parallel_include.o set_mpi_comm_4_solvers.o util_param.o <br>
> thread_util.o<br>
> ar: creating libutil.a<br>
> a - clocks_handler.o<br>
> a - device_helper.o<br>
> a - divide.o<br>
> a - data_buffer.o<br>
> a - error_handler.o<br>
> a - export_gstart_2_solvers.o<br>
> a - find_free_unit.o<br>
> a - fletcher32_mod.o<br>
> a - mem_counter.o<br>
> a - mp.o<br>
> a - mp_base.o<br>
> a - mp_base_gpu.o<br>
> a - mp_bands_util.o<br>
> a - parallel_include.o<br>
> a - set_mpi_comm_4_solvers.o<br>
> a - util_param.o<br>
> a - thread_util.o<br>
> ranlib libutil.a<br>
> make[1]: Leaving directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/UtilXlib'<br>
> cd install ; make -f extlibs_makefile liblapack<br>
> make[1]: Entering directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/install'<br>
> make[1]: Nothing to be done for `liblapack'.<br>
> make[1]: Leaving directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/install'<br>
> cd install ; make -f extlibs_makefile libcuda<br>
> make[1]: Entering directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/install'<br>
> cd ../external/devxlib; \<br>
> if test ! -e configure; then \<br>
> wget <br>
> "<a href="https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz" rel="noreferrer" target="_blank">https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz</a> <br>
> <<a href="https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz" rel="noreferrer" target="_blank">https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz</a>>" <br>
> -O devxlib.tar.gz || curl <br>
> "<a href="https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz" rel="noreferrer" target="_blank">https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz</a> <br>
> <<a href="https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz" rel="noreferrer" target="_blank">https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz</a>>" <br>
> -o devxlib.tar.gz ; \<br>
> tar xzf devxlib.tar.gz --strip-components=1 -C . ; \<br>
> rm devxlib.tar.gz ; \<br>
> fi; \<br>
> touch make.inc; \<br>
> make clean; \<br>
> export F90FLAGS=""; \<br>
> ./configure FC=nvfortran CC=pgcc \<br>
> --with-cuda= \<br>
> --with-cuda-cc=70 \<br>
> --with-cuda-runtime=11.0 \<br>
> --disable-parallel \<br>
> --enable-cuda-env-check=no; \<br>
> make all<br>
> --2020-12-19 09:29:12-- <br>
> <a href="https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz" rel="noreferrer" target="_blank">https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz</a> <br>
> <<a href="https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz" rel="noreferrer" target="_blank">https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz</a>><br>
> Resolving <a href="http://gitlab.com" rel="noreferrer" target="_blank">gitlab.com</a> <br>
> <<a href="http://gitlab.com/" rel="noreferrer" target="_blank">http://gitlab.com/</a>> <br>
> (<a href="http://gitlab.com" rel="noreferrer" target="_blank">gitlab.com</a> <br>
> <<a href="http://gitlab.com/" rel="noreferrer" target="_blank">http://gitlab.com/</a>>)... <br>
> 172.65.251.78, 2606:4700:90:0:f22e:fbec:5bed:a9b9<br>
> Connecting to <a href="http://gitlab.com" rel="noreferrer" target="_blank">gitlab.com</a> <br>
> <<a href="http://gitlab.com/" rel="noreferrer" target="_blank">http://gitlab.com/</a>> <br>
> (<a href="http://gitlab.com" rel="noreferrer" target="_blank">gitlab.com</a> <br>
> <<a href="http://gitlab.com/" rel="noreferrer" target="_blank">http://gitlab.com/</a>>)|172.65.251.78|:443... <br>
> connected.<br>
> HTTP request sent, awaiting response... 403 Forbidden<br>
> 2020-12-19 09:29:23 ERROR 403: Forbidden.<br>
> <br>
> % Total % Received % Xferd Average Speed Time Time Time <br>
> Current<br>
> Dload Upload Total Spent Left <br>
> Speed<br>
> 100 16 100 16 0 0 27 0 --:--:-- --:--:-- --:--:-- <br>
> 27<br>
> <br>
> gzip: stdin: not in gzip format<br>
> tar: Child returned status 1<br>
> tar: Error is not recoverable: exiting now<br>
> make[2]: Entering directory <br>
> `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/external/devxlib'<br>
> make[2]: *** No rule to make target `clean'. Stop.<br>
> make[2]: Leaving directory <br>
> `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/external/devxlib'<br>
> /bin/sh: line 9: ./configure: No such file or directory<br>
> make[2]: Entering directory <br>
> `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/external/devxlib'<br>
> make[2]: *** No rule to make target `all'. Stop.<br>
> make[2]: Leaving directory <br>
> `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/external/devxlib'<br>
> make[1]: *** [libcuda_devxlib] Error 2<br>
> make[1]: Leaving directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/install'<br>
> make: *** [libcuda] Error 2<br>
> <br>
> *Moreover, "which libcuda.so" results in:*<br>
> *<br>
> *<br>
> /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.0/lib64/stubs/libcuda.so<br>
> *<br>
> *<br>
> *Any help will be greatly appreciated.*<br>
> *<br>
> *<br>
> *Best regards,*<br>
> *<br>
> *<br>
> *Mohammad*<br>
> *ShirazU*<br>
> <br>
> On Wed, Dec 16, 2020 at 5:08 PM Pietro Bonfa' <<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a> <br>
> <mailto:<a href="mailto:pietro.bonfa@unipr.it" target="_blank">pietro.bonfa@unipr.it</a>>> wrote:<br>
> <br>
> Dear Mohammad,<br>
> <br>
> CUDA may be installed somewhere else, anyway, if you want to skip the<br>
> environment check, you may configure QE with this command<br>
> <br>
> ./configure FC=pgf90 F90=pgf90 CC=pgcc --with-cuda=yes<br>
> --enable-cuda-env-check=no --with-cuda-runtime=SETME --with-cuda-cc=70<br>
> --enable-openmp<br>
> <br>
> Remember to set the cuda runtime according to what is provided by your<br>
> setup.<br>
> <br>
> Hope this helps,<br>
> best,<br>
> Pietro<br>
> <br>
> ---<br>
> Pietro Bonfà<br>
> University of Parma<br>
> <br>
> <br>
> On 12/16/20 9:10 AM, Mohammad Moaddeli wrote:<br>
> > Dear all,<br>
> > I am trying to compile the 6.7 version of the code using PGI 2020.<br>
> > I followed these steps:<br>
> ><br>
> > *1) NVIDIA driver (NVIDIA-Linux-x86_64-450.80.02.rpm) is installed.*<br>
> > *the output of nvidia-smi:*<br>
> ><br>
> > Wed Dec 16 09:07:11 2020<br>
> ><br>
> +-----------------------------------------------------------------------------+<br>
> > | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version:<br>
> > 11.0 |<br>
> ><br>
> |-------------------------------+----------------------+----------------------+<br>
> > | GPU Name Persistence-M| Bus-Id Disp.A | Volatile<br>
> > Uncorr. ECC |<br>
> > | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util<br>
> > Compute M. |<br>
> > | | |<br>
> > MIG M. |<br>
> ><br>
> |===============================+======================+======================|<br>
> > | 0 TITAN V Off | 00000000:06:00.0 Off |<br>
> > N/A |<br>
> > | 27% 37C P0 32W / 250W | 0MiB / 12066MiB | 0%<br>
> > Default |<br>
> > | | |<br>
> > N/A |<br>
> ><br>
> +-------------------------------+----------------------+----------------------+<br>
> > | 1 TITAN V Off | 00000000:07:00.0 Off |<br>
> > N/A |<br>
> > | 25% 37C P0 35W / 250W | 0MiB / 12066MiB | 0%<br>
> > Default |<br>
> > | | |<br>
> > N/A |<br>
> ><br>
> +-------------------------------+----------------------+----------------------+<br>
> ><br>
> ><br>
> +-----------------------------------------------------------------------------+<br>
> > | Processes:<br>
> > |<br>
> > | GPU GI CI PID Type Process name <br>
> GPU<br>
> > Memory |<br>
> > | ID ID <br>
> Usage<br>
> > |<br>
> ><br>
> |=============================================================================|<br>
> > | No running processes found<br>
> > |<br>
> ><br>
> +-----------------------------------------------------------------------------+<br>
> ><br>
> > *The output of pgaccelinfo:*<br>
> ><br>
> > CUDA Driver Version: 11000<br>
> > NVRM version: NVIDIA UNIX x86_64 Kernel Module<br>
> > 450.80.02 Wed Sep 23 01:13:39 UTC 2020<br>
> ><br>
> > Device Number: 0<br>
> > Device Name: TITAN V<br>
> > Device Revision Number: 7.0<br>
> > Global Memory Size: 12652838912<br>
> > Number of Multiprocessors: 80<br>
> > Concurrent Copy and Execution: Yes<br>
> > Total Constant Memory: 65536<br>
> > Total Shared Memory per Block: 49152<br>
> > Registers per Block: 65536<br>
> > Warp Size: 32<br>
> > Maximum Threads per Block: 1024<br>
> > Maximum Block Dimensions: 1024, 1024, 64<br>
> > Maximum Grid Dimensions: 2147483647 x 65535 x 65535<br>
> > Maximum Memory Pitch: 2147483647B<br>
> > Texture Alignment: 512B<br>
> > Clock Rate: 1455 MHz<br>
> > Execution Timeout: No<br>
> > Integrated Device: No<br>
> > Can Map Host Memory: Yes<br>
> > Compute Mode: default<br>
> > Concurrent Kernels: Yes<br>
> > ECC Enabled: No<br>
> > Memory Clock Rate: 850 MHz<br>
> > Memory Bus Width: 3072 bits<br>
> > L2 Cache Size: 4718592 bytes<br>
> > Max Threads Per SMP: 2048<br>
> > Async Engines: 7<br>
> > Unified Addressing: Yes<br>
> > Managed Memory: Yes<br>
> > Concurrent Managed Memory: Yes<br>
> > Preemption Supported: Yes<br>
> > Cooperative Launch: Yes<br>
> > Multi-Device: Yes<br>
> > Default Target: cc70<br>
> ><br>
> > Device Number: 1<br>
> > Device Name: TITAN V<br>
> > Device Revision Number: 7.0<br>
> > Global Memory Size: 12652838912<br>
> > Number of Multiprocessors: 80<br>
> > Concurrent Copy and Execution: Yes<br>
> > Total Constant Memory: 65536<br>
> > Total Shared Memory per Block: 49152<br>
> > Registers per Block: 65536<br>
> > Warp Size: 32<br>
> > Maximum Threads per Block: 1024<br>
> > Maximum Block Dimensions: 1024, 1024, 64<br>
> > Maximum Grid Dimensions: 2147483647 x 65535 x 65535<br>
> > Maximum Memory Pitch: 2147483647B<br>
> > Texture Alignment: 512B<br>
> > Clock Rate: 1455 MHz<br>
> > Execution Timeout: No<br>
> > Integrated Device: No<br>
> > Can Map Host Memory: Yes<br>
> > Compute Mode: default<br>
> > Concurrent Kernels: Yes<br>
> > ECC Enabled: No<br>
> > Memory Clock Rate: 850 MHz<br>
> > Memory Bus Width: 3072 bits<br>
> > L2 Cache Size: 4718592 bytes<br>
> > Max Threads Per SMP: 2048<br>
> > Async Engines: 7<br>
> > Unified Addressing: Yes<br>
> > Managed Memory: Yes<br>
> > Concurrent Managed Memory: Yes<br>
> > Preemption Supported: Yes<br>
> > Cooperative Launch: Yes<br>
> > Multi-Device: Yes<br>
> > Default Target: cc70<br>
> ><br>
> > *2) PGI compiler is installed:*<br>
> > *yum install nvhpc-20-11-20.11-1.x86_64.rpm<br>
> nvhpc-2020-20.11-1.x86_64.rpm<br>
> > *<br>
> > *PATHs that are set in ~/.bashrc file:<br>
> > *<br>
> > *<br>
> > *<br>
> > export<br>
> PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/bin:$PATH<br>
> > export<br>
> PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/include:$PATH<br>
> > export<br>
> ><br>
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/lib64:$LD_LIBRARY_PATH<br>
> > export<br>
> ><br>
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/extras/CUPTI/lib64:$LD_LIBRARY_PATH<br>
> > export<br>
> ><br>
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/lib64/stubs:$LD_LIBRARY_PATH<br>
> > NVARCH=`uname -s`_`uname -m`; export NVARCH<br>
> > NVCOMPILERS=/opt/nvidia/hpc_sdk; export NVCOMPILERS<br>
> > MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/20.11/compilers/man; export<br>
> MANPATH<br>
> > PATH=$NVCOMPILERS/$NVARCH/20.11/compilers/bin:$PATH; export PATH<br>
> > PATH=$NVCOMPILERS/$NVARCH/20.11/compilers/include:$PATH; export PATH<br>
> > LD_LIBRARY_PATH=$NVCOMPILERS/$NVARCH/20.11/compilers/lib:$PATH;<br>
> export<br>
> > LD_LIBRARY_PATH<br>
> > export<br>
> PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/comm_libs/mpi/bin:$PATH<br>
> > export<br>
> ><br>
> PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/comm_libs/mpi/include:$PATH<br>
> > export<br>
> ><br>
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/comm_libs/mpi/lib:$LD_LIBRARY_PATH<br>
> > export<br>
> ><br>
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/math_libs/11.1/lib64:$LD_LIBRARY_PATH<br>
> > export<br>
> ><br>
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/math_libs/11.1/lib64/stubs:$LD_LIBRARY_PATH*<br>
> > *<br>
> > *<br>
> > *<br>
> > *3) compiling the code using:*<br>
> > *./configure FC=pgf90 F90=pgf90 F77=pgf90 CC=pgcc MPIF90=mpif90<br>
> > --with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda<br>
> > --with-cuda-runtime=11.1 --with-cuda-cc=70 --enable-openmp<br>
> > --with-scalapack=no*<br>
> ><br>
> > checking build system type... x86_64-pc-linux-gnu<br>
> > checking ARCH... x86_64<br>
> > checking setting AR... ... ar<br>
> > checking setting ARFLAGS... ... ruv<br>
> > checking whether the Fortran compiler works... yes<br>
> > checking for Fortran compiler default output file name... a.out<br>
> > checking for suffix of executables...<br>
> > checking whether we are cross compiling... no<br>
> > checking for suffix of object files... o<br>
> > checking whether we are using the GNU Fortran compiler... no<br>
> > checking whether pgf90 accepts -g... yes<br>
> > configure: WARNING: F90 value is set to be consistent with value<br>
> of MPIF90<br>
> > checking for mpif90... mpif90<br>
> > checking whether we are using the GNU Fortran compiler... no<br>
> > checking whether mpif90 accepts -g... yes<br>
> > checking version of mpif90... nvfortran 20.11-0<br>
> > checking for Fortran flag to compile .f90 files... none<br>
> > setting F90... nvfortran<br>
> > setting MPIF90... mpif90<br>
> > checking whether we are using the GNU C compiler... yes<br>
> > checking whether pgcc accepts -g... yes<br>
> > checking for pgcc option to accept ISO C89... none needed<br>
> > setting CC... pgcc<br>
> > setting CFLAGS... -fast -Mpreprocess<br>
> > using F90... nvfortran<br>
> > setting FFLAGS... -O1<br>
> > setting F90FLAGS... $(FFLAGS)<br>
> > setting FFLAGS_NOOPT... -O0<br>
> > setting CPP... cpp<br>
> > setting CPPFLAGS... -P -traditional -Uvector<br>
> > setting LD... mpif90<br>
> > setting LDFLAGS...<br>
> > checking for Fortran flag to compile .f90 files... (cached) none<br>
> > checking whether Fortran compiler accepts -Mcuda=cuda11.1... yes<br>
> > checking for nvcc...<br>
> > /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvcc<br>
> > checking whether nvcc works... no<br>
> > configure: WARNING: CUDA compiler has problems.<br>
> > checking for cuInit in -lcuda... no<br>
> > configure: error: in `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7':<br>
> > configure: error: Couldn't find libcuda<br>
> > See `config.log' for more details*<br>
> > *<br>
> > *<br>
> > *<br>
> > *<br>
> > *<br>
> > *<br>
> > *<br>
> > *Any Help will be greatly appreciated.*<br>
> > *<br>
> > *<br>
> > *<br>
> > *<br>
> > *<br>
> > *<br>
> > *P.S.*<br>
> > *When I run nvcc in terminal, the following error appears:*<br>
> > $ which nvcc<br>
> > /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvcc<br>
> > **<br>
> > $ nvcc<br>
> > nvcc-Error-CUDA version 10.2 was not installed with this HPC SDK:<br>
> > /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/10.2/bin<br>
> ><br>
> ><br>
> > *Best,*<br>
> > *Mohammad Moaddeli*<br>
> > *Shiraz University*<br>
> > **<br>
> > **<br>
> ><br>
> > _______________________________________________<br>
> > Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu" rel="noreferrer" target="_blank">www.max-centre.eu</a><br>
> <<a href="http://www.max-centre.eu/" rel="noreferrer" target="_blank">http://www.max-centre.eu/</a>>)<br>
> > users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> > <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> <<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a>><br>
> ><br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu" rel="noreferrer" target="_blank">www.max-centre.eu</a><br>
> <<a href="http://www.max-centre.eu/" rel="noreferrer" target="_blank">http://www.max-centre.eu/</a>>)<br>
> users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
> <mailto:<a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a>><br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> <<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a>><br>
> <br>
> <br>
> _______________________________________________<br>
> Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu" rel="noreferrer" target="_blank">www.max-centre.eu</a>)<br>
> users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
> <a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a><br>
> <br>
_______________________________________________<br>
Quantum ESPRESSO is supported by MaX (<a href="http://www.max-centre.eu" rel="noreferrer" target="_blank">www.max-centre.eu</a>)<br>
users mailing list <a href="mailto:users@lists.quantum-espresso.org" target="_blank">users@lists.quantum-espresso.org</a><br>
<a href="https://lists.quantum-espresso.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.quantum-espresso.org/mailman/listinfo/users</a></blockquote></div>