[QE-users] [ge-GPU] compiling q-e-gpu-qe-gpu-6.7
Mohammad Moaddeli
mohammad.moaddeli at gmail.com
Sat Dec 19 07:27:34 CET 2020
Dear Louis and Pietro,
*With the config options as following:*
*./configure FC=pgf90 F90=pgf90 CC=pgcc
--with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.0
--with-cuda-runtime=11.0 --with-cuda-cc=70 --enable-openmp
--with-scalapack=no*
*results is:*
checking build system type... x86_64-pc-linux-gnu
checking ARCH... x86_64
checking setting AR... ... ar
checking setting ARFLAGS... ... ruv
checking whether the Fortran compiler works... yes
checking for Fortran compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU Fortran compiler... no
checking whether pgf90 accepts -g... yes
configure: WARNING: F90 value is set to be consistent with value of MPIF90
checking for mpiifort... no
checking for mpif90... mpif90
checking whether we are using the GNU Fortran compiler... no
checking whether mpif90 accepts -g... yes
checking version of mpif90... nvfortran 20.11-0
checking for Fortran flag to compile .f90 files... none
setting F90... nvfortran
setting MPIF90... mpif90
checking whether we are using the GNU C compiler... yes
checking whether pgcc accepts -g... yes
checking for pgcc option to accept ISO C89... none needed
setting CC... pgcc
setting CFLAGS... -fast -Mpreprocess
using F90... nvfortran
setting FFLAGS... -O1
setting F90FLAGS... $(FFLAGS)
setting FFLAGS_NOOPT... -O0
setting CPP... cpp
setting CPPFLAGS... -P -traditional -Uvector
setting LD... mpif90
setting LDFLAGS...
checking for Fortran flag to compile .f90 files... (cached) none
checking whether Fortran compiler accepts -Mcuda=cuda11.0... yes
checking for nvcc...
/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvcc
checking whether nvcc works... yes
checking for cuInit in -lcuda... no
configure: error: in `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7':
configure: error: Couldn't find libcuda
See `config.log' for more details
*Although the option --enable-cuda-env-check=no resulted in the
configuration finished:*
*./configure FC=pgf90 F90=pgf90 CC=pgcc --with-cuda=yes
--enable-cuda-env-check=no --with-cuda-runtime=11.0 --with-cuda-cc=70
--enable-openmp --with-scalapack=no*
checking build system type... x86_64-pc-linux-gnu
checking ARCH... x86_64
checking setting AR... ... ar
checking setting ARFLAGS... ... ruv
checking whether the Fortran compiler works... yes
checking for Fortran compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU Fortran compiler... no
checking whether pgf90 accepts -g... yes
configure: WARNING: F90 value is set to be consistent with value of MPIF90
checking for mpiifort... no
checking for mpif90... mpif90
checking whether we are using the GNU Fortran compiler... no
checking whether mpif90 accepts -g... yes
checking version of mpif90... nvfortran 20.11-0
checking for Fortran flag to compile .f90 files... none
setting F90... nvfortran
setting MPIF90... mpif90
checking whether we are using the GNU C compiler... yes
checking whether pgcc accepts -g... yes
checking for pgcc option to accept ISO C89... none needed
setting CC... pgcc
setting CFLAGS... -fast -Mpreprocess
using F90... nvfortran
setting FFLAGS... -O1
setting F90FLAGS... $(FFLAGS)
setting FFLAGS_NOOPT... -O0
setting CPP... cpp
setting CPPFLAGS... -P -traditional -Uvector
setting LD... mpif90
setting LDFLAGS...
checking for Fortran flag to compile .f90 files... (cached) none
checking whether Fortran compiler accepts -Mcuda=cuda11.0... yes
checking for /usr/local/cuda/... no
checking for /usr/local/cuda/include... no
checking for /usr/local/cuda/lib64... no
checking for nvcc...
/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvcc
checking whether nvcc works... yes
checking for cusolverDnZhegvdx_bufferSize in -lcusolver... no
configure: WARNING: Using legacy custom solver.
checking whether make sets $(MAKE)... yes
checking whether Fortran files must be preprocessed... yes
checking whether we are using the GNU Fortran 77 compiler... no
checking whether nvfortran accepts -g... yes
checking for library containing dgemm... no
MKL not found
in /opt/intel/mkl/lib/intel64: checking for library containing dgemm...
-lmkl_intel_lp64
setting BLAS_LIBS... -L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64
-lmkl_intel_thread -lmkl_core
checking FFT...
checking MASS...
checking for library containing mpi_init... none required
checking ELPA...
checking BEEF... -lbeef
setting BEEF_LIBS... $(TOPDIR)/LIBBEEF/libbeef.a
checking for ranlib... ranlib
checking for wget... wget -O
setting WGET... wget -O
setting DFLAGS... -D__CUDA -D__DFTI -D__MPI
setting IFLAGS... -I$(TOPDIR)/include -I$(TOPDIR)/FoX/finclude
-I/opt/intel/mkl/include
configure: creating ./config.status
config.status: creating install/make_lapack.inc
config.status: creating include/configure.h
config.status: creating make.inc
config.status: creating configure.msg
config.status: creating install/make_wannier90.inc
config.status: creating include/qe_cdefs.h
--------------------------------------------------------------------
ESPRESSO can take advantage of several optimized numerical libraries
(essl, fftw, mkl...). This configure script attempts to find them,
but may fail if they have been installed in non-standard locations.
If a required library is not found, the local copy will be compiled.
The following libraries have been found:
BLAS_LIBS=-L/opt/intel/mkl/lib/intel64 -lmkl_intel_lp64
-lmkl_intel_thread -lmkl_core
LAPACK_LIBS=
FFT_LIBS=
Please check if this is what you expect.
If any libraries are missing, you may specify a list of directories
to search and retry, as follows:
./configure LIBDIRS="list of directories, separated by spaces"
Parallel environment detected successfully.\
Configured for compilation of parallel executables.
For more info, read the ESPRESSO User's Guide (Doc/users-guide.tex).
--------------------------------------------------------------------
configure: success
*make pw results in:*
....
.....
....
cpp -P -traditional -Uvector -D__CUDA -D__DFTI -D__MPI
-I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//include
-I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//FoX/finclude
-I/opt/intel/mkl/include thread_util.f90 -o thread_util_tmp.f90 ; \
mpif90 -O1 -D__CUDA -D__DFTI -D__MPI -Mcuda=cc70,cuda11.0
-I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//external/devxlib/src
-I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//external/devxlib/include
-I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//EIGENSOLVER_GPU/lib_eigsolve
-I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//external/devxlib/src
-I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//include
-I/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7//FoX/finclude
-I/opt/intel/mkl/include -I. -c thread_util_tmp.f90 -o thread_util.o
ar ruv libutil.a clocks_handler.o device_helper.o divide.o data_buffer.o
error_handler.o export_gstart_2_solvers.o find_free_unit.o fletcher32_mod.o
mem_counter.o mp.o mp_base.o mp_base_gpu.o mp_bands_util.o
parallel_include.o set_mpi_comm_4_solvers.o util_param.o thread_util.o
ar: creating libutil.a
a - clocks_handler.o
a - device_helper.o
a - divide.o
a - data_buffer.o
a - error_handler.o
a - export_gstart_2_solvers.o
a - find_free_unit.o
a - fletcher32_mod.o
a - mem_counter.o
a - mp.o
a - mp_base.o
a - mp_base_gpu.o
a - mp_bands_util.o
a - parallel_include.o
a - set_mpi_comm_4_solvers.o
a - util_param.o
a - thread_util.o
ranlib libutil.a
make[1]: Leaving directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/UtilXlib'
cd install ; make -f extlibs_makefile liblapack
make[1]: Entering directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/install'
make[1]: Nothing to be done for `liblapack'.
make[1]: Leaving directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/install'
cd install ; make -f extlibs_makefile libcuda
make[1]: Entering directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/install'
cd ../external/devxlib; \
if test ! -e configure; then \
wget "
https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz"
-O devxlib.tar.gz || curl "
https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz"
-o devxlib.tar.gz ; \
tar xzf devxlib.tar.gz --strip-components=1 -C . ; \
rm devxlib.tar.gz ; \
fi; \
touch make.inc; \
make clean; \
export F90FLAGS=""; \
./configure FC=nvfortran CC=pgcc \
--with-cuda= \
--with-cuda-cc=70 \
--with-cuda-runtime=11.0 \
--disable-parallel \
--enable-cuda-env-check=no; \
make all
--2020-12-19 09:29:12--
https://gitlab.com/max-centre/components/devicexlib/-/archive/master/devicexlib-master.tar.gz
Resolving gitlab.com (gitlab.com)... 172.65.251.78,
2606:4700:90:0:f22e:fbec:5bed:a9b9
Connecting to gitlab.com (gitlab.com)|172.65.251.78|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2020-12-19 09:29:23 ERROR 403: Forbidden.
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left
Speed
100 16 100 16 0 0 27 0 --:--:-- --:--:-- --:--:--
27
gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
make[2]: Entering directory
`/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/external/devxlib'
make[2]: *** No rule to make target `clean'. Stop.
make[2]: Leaving directory
`/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/external/devxlib'
/bin/sh: line 9: ./configure: No such file or directory
make[2]: Entering directory
`/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/external/devxlib'
make[2]: *** No rule to make target `all'. Stop.
make[2]: Leaving directory
`/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/external/devxlib'
make[1]: *** [libcuda_devxlib] Error 2
make[1]: Leaving directory `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7/install'
make: *** [libcuda] Error 2
*Moreover, "which libcuda.so" results in:*
/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.0/lib64/stubs/libcuda.so
*Any help will be greatly appreciated.*
*Best regards,*
*Mohammad*
*ShirazU*
On Wed, Dec 16, 2020 at 5:08 PM Pietro Bonfa' <pietro.bonfa at unipr.it> wrote:
> Dear Mohammad,
>
> CUDA may be installed somewhere else, anyway, if you want to skip the
> environment check, you may configure QE with this command
>
> ./configure FC=pgf90 F90=pgf90 CC=pgcc --with-cuda=yes
> --enable-cuda-env-check=no --with-cuda-runtime=SETME --with-cuda-cc=70
> --enable-openmp
>
> Remember to set the cuda runtime according to what is provided by your
> setup.
>
> Hope this helps,
> best,
> Pietro
>
> ---
> Pietro Bonfà
> University of Parma
>
>
> On 12/16/20 9:10 AM, Mohammad Moaddeli wrote:
> > Dear all,
> > I am trying to compile the 6.7 version of the code using PGI 2020.
> > I followed these steps:
> >
> > *1) NVIDIA driver (NVIDIA-Linux-x86_64-450.80.02.rpm) is installed.*
> > *the output of nvidia-smi:*
> >
> > Wed Dec 16 09:07:11 2020
> >
> +-----------------------------------------------------------------------------+
> > | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version:
> > 11.0 |
> >
> |-------------------------------+----------------------+----------------------+
> > | GPU Name Persistence-M| Bus-Id Disp.A | Volatile
> > Uncorr. ECC |
> > | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util
> > Compute M. |
> > | | |
> > MIG M. |
> >
> |===============================+======================+======================|
> > | 0 TITAN V Off | 00000000:06:00.0 Off |
> > N/A |
> > | 27% 37C P0 32W / 250W | 0MiB / 12066MiB | 0%
> > Default |
> > | | |
> > N/A |
> >
> +-------------------------------+----------------------+----------------------+
> > | 1 TITAN V Off | 00000000:07:00.0 Off |
> > N/A |
> > | 25% 37C P0 35W / 250W | 0MiB / 12066MiB | 0%
> > Default |
> > | | |
> > N/A |
> >
> +-------------------------------+----------------------+----------------------+
> >
> >
> +-----------------------------------------------------------------------------+
> > | Processes:
> > |
> > | GPU GI CI PID Type Process name GPU
> > Memory |
> > | ID ID Usage
> > |
> >
> |=============================================================================|
> > | No running processes found
> > |
> >
> +-----------------------------------------------------------------------------+
> >
> > *The output of pgaccelinfo:*
> >
> > CUDA Driver Version: 11000
> > NVRM version: NVIDIA UNIX x86_64 Kernel Module
> > 450.80.02 Wed Sep 23 01:13:39 UTC 2020
> >
> > Device Number: 0
> > Device Name: TITAN V
> > Device Revision Number: 7.0
> > Global Memory Size: 12652838912
> > Number of Multiprocessors: 80
> > Concurrent Copy and Execution: Yes
> > Total Constant Memory: 65536
> > Total Shared Memory per Block: 49152
> > Registers per Block: 65536
> > Warp Size: 32
> > Maximum Threads per Block: 1024
> > Maximum Block Dimensions: 1024, 1024, 64
> > Maximum Grid Dimensions: 2147483647 x 65535 x 65535
> > Maximum Memory Pitch: 2147483647B
> > Texture Alignment: 512B
> > Clock Rate: 1455 MHz
> > Execution Timeout: No
> > Integrated Device: No
> > Can Map Host Memory: Yes
> > Compute Mode: default
> > Concurrent Kernels: Yes
> > ECC Enabled: No
> > Memory Clock Rate: 850 MHz
> > Memory Bus Width: 3072 bits
> > L2 Cache Size: 4718592 bytes
> > Max Threads Per SMP: 2048
> > Async Engines: 7
> > Unified Addressing: Yes
> > Managed Memory: Yes
> > Concurrent Managed Memory: Yes
> > Preemption Supported: Yes
> > Cooperative Launch: Yes
> > Multi-Device: Yes
> > Default Target: cc70
> >
> > Device Number: 1
> > Device Name: TITAN V
> > Device Revision Number: 7.0
> > Global Memory Size: 12652838912
> > Number of Multiprocessors: 80
> > Concurrent Copy and Execution: Yes
> > Total Constant Memory: 65536
> > Total Shared Memory per Block: 49152
> > Registers per Block: 65536
> > Warp Size: 32
> > Maximum Threads per Block: 1024
> > Maximum Block Dimensions: 1024, 1024, 64
> > Maximum Grid Dimensions: 2147483647 x 65535 x 65535
> > Maximum Memory Pitch: 2147483647B
> > Texture Alignment: 512B
> > Clock Rate: 1455 MHz
> > Execution Timeout: No
> > Integrated Device: No
> > Can Map Host Memory: Yes
> > Compute Mode: default
> > Concurrent Kernels: Yes
> > ECC Enabled: No
> > Memory Clock Rate: 850 MHz
> > Memory Bus Width: 3072 bits
> > L2 Cache Size: 4718592 bytes
> > Max Threads Per SMP: 2048
> > Async Engines: 7
> > Unified Addressing: Yes
> > Managed Memory: Yes
> > Concurrent Managed Memory: Yes
> > Preemption Supported: Yes
> > Cooperative Launch: Yes
> > Multi-Device: Yes
> > Default Target: cc70
> >
> > *2) PGI compiler is installed:*
> > *yum install nvhpc-20-11-20.11-1.x86_64.rpm nvhpc-2020-20.11-1.x86_64.rpm
> > *
> > *PATHs that are set in ~/.bashrc file:
> > *
> > *
> > *
> > export PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/bin:$PATH
> > export
> PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/include:$PATH
> > export
> >
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/lib64:$LD_LIBRARY_PATH
> > export
> >
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/extras/CUPTI/lib64:$LD_LIBRARY_PATH
> > export
> >
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/11.1/lib64/stubs:$LD_LIBRARY_PATH
> > NVARCH=`uname -s`_`uname -m`; export NVARCH
> > NVCOMPILERS=/opt/nvidia/hpc_sdk; export NVCOMPILERS
> > MANPATH=$MANPATH:$NVCOMPILERS/$NVARCH/20.11/compilers/man; export MANPATH
> > PATH=$NVCOMPILERS/$NVARCH/20.11/compilers/bin:$PATH; export PATH
> > PATH=$NVCOMPILERS/$NVARCH/20.11/compilers/include:$PATH; export PATH
> > LD_LIBRARY_PATH=$NVCOMPILERS/$NVARCH/20.11/compilers/lib:$PATH; export
> > LD_LIBRARY_PATH
> > export
> PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/comm_libs/mpi/bin:$PATH
> > export
> > PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/comm_libs/mpi/include:$PATH
> > export
> >
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/comm_libs/mpi/lib:$LD_LIBRARY_PATH
> > export
> >
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/math_libs/11.1/lib64:$LD_LIBRARY_PATH
> > export
> >
> LD_LIBRARY_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/math_libs/11.1/lib64/stubs:$LD_LIBRARY_PATH*
> > *
> > *
> > *
> > *3) compiling the code using:*
> > *./configure FC=pgf90 F90=pgf90 F77=pgf90 CC=pgcc MPIF90=mpif90
> > --with-cuda=/opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda
> > --with-cuda-runtime=11.1 --with-cuda-cc=70 --enable-openmp
> > --with-scalapack=no*
> >
> > checking build system type... x86_64-pc-linux-gnu
> > checking ARCH... x86_64
> > checking setting AR... ... ar
> > checking setting ARFLAGS... ... ruv
> > checking whether the Fortran compiler works... yes
> > checking for Fortran compiler default output file name... a.out
> > checking for suffix of executables...
> > checking whether we are cross compiling... no
> > checking for suffix of object files... o
> > checking whether we are using the GNU Fortran compiler... no
> > checking whether pgf90 accepts -g... yes
> > configure: WARNING: F90 value is set to be consistent with value of
> MPIF90
> > checking for mpif90... mpif90
> > checking whether we are using the GNU Fortran compiler... no
> > checking whether mpif90 accepts -g... yes
> > checking version of mpif90... nvfortran 20.11-0
> > checking for Fortran flag to compile .f90 files... none
> > setting F90... nvfortran
> > setting MPIF90... mpif90
> > checking whether we are using the GNU C compiler... yes
> > checking whether pgcc accepts -g... yes
> > checking for pgcc option to accept ISO C89... none needed
> > setting CC... pgcc
> > setting CFLAGS... -fast -Mpreprocess
> > using F90... nvfortran
> > setting FFLAGS... -O1
> > setting F90FLAGS... $(FFLAGS)
> > setting FFLAGS_NOOPT... -O0
> > setting CPP... cpp
> > setting CPPFLAGS... -P -traditional -Uvector
> > setting LD... mpif90
> > setting LDFLAGS...
> > checking for Fortran flag to compile .f90 files... (cached) none
> > checking whether Fortran compiler accepts -Mcuda=cuda11.1... yes
> > checking for nvcc...
> > /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvcc
> > checking whether nvcc works... no
> > configure: WARNING: CUDA compiler has problems.
> > checking for cuInit in -lcuda... no
> > configure: error: in `/codes/qe_6.7_GPU/q-e-gpu-qe-gpu-6.7':
> > configure: error: Couldn't find libcuda
> > See `config.log' for more details*
> > *
> > *
> > *
> > *
> > *
> > *
> > *
> > *Any Help will be greatly appreciated.*
> > *
> > *
> > *
> > *
> > *
> > *
> > *P.S.*
> > *When I run nvcc in terminal, the following error appears:*
> > $ which nvcc
> > /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/compilers/bin/nvcc
> > **
> > $ nvcc
> > nvcc-Error-CUDA version 10.2 was not installed with this HPC SDK:
> > /opt/nvidia/hpc_sdk/Linux_x86_64/20.11/cuda/10.2/bin
> >
> >
> > *Best,*
> > *Mohammad Moaddeli*
> > *Shiraz University*
> > **
> > **
> >
> > _______________________________________________
> > Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
> > users mailing list users at lists.quantum-espresso.org
> > https://lists.quantum-espresso.org/mailman/listinfo/users
> >
> _______________________________________________
> Quantum ESPRESSO is supported by MaX (www.max-centre.eu)
> users mailing list users at lists.quantum-espresso.org
> https://lists.quantum-espresso.org/mailman/listinfo/users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.quantum-espresso.org/pipermail/users/attachments/20201219/9ace30ee/attachment.html>
More information about the users
mailing list