[QE-users] [QE 7.0/GPU] Build configured with Autotools fails
Robert MIJAKOVIC
robert.mijakovic at lxp.lu
Tue May 24 17:37:10 CEST 2022
Hi Paolo,
I will have a look more closely to the links you sent me.
I tried to specify the location of fftw3.f03 file by setting INCLUDE_FFTW variable to the location of our FFTW library, compiled with NVHPC.
Unfortunately, it didn't work.
I have checked the m4 macro from the QE-7.0 used to detect the location of FFTW, x_ac_qe_fft.m4, and compared it with the tip of the develop branch on gitlab.
The only thing that differs is the FFLAGS variable, -ffree-form option is added (FFLAGS="$orig_fflags -I$inc -ffree-form").
This means that either Gitlab doesn't provide the latest development version or that the improved FFTW3 auto-detection macro is available in the 7.0 release tarball.
I have added the location of FFTW include directory directly to make.inc. This helped to continue the compilation process until:
mpif90 -mp -cuda -gpu=cc80,cuda11.0 -acc -o pw.x \
pwscf.o libpw.a ../../KS_Solvers/libks_solvers.a ../../dft-d3/libdftd3qe.a /mnt/tier2/users/apps/scratchpad/qe-7.0//Modules/libqemod.a /mnt/tier2/users/apps/scratchpad/qe-7.0//upflib/libupf.a /mnt/tier2/users/apps/scratchpad/qe-7.0//XClib/xc_lib.a /mnt/tier2/users/apps/scratchpad/qe-7.0//FFTXlib/libqefft.a /mnt/tier2/users/apps/scratchpad/qe-7.0//LAXlib/libqela.a /mnt/tier2/users/apps/scratchpad/qe-7.0//UtilXlib/libutil.a /mnt/tier2/users/apps/scratchpad/qe-7.0//MBD/libmbd.a -cudalib=cufft,cublas,cusolver,curand /mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src/libdevXlib.a -L/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -ldevXlib -L/apps/USE/easybuild/staging/2021.1/software/NVHPC/22.3/Linux_x86_64/22.3/compilers/lib -llapack -L/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/lib -lFoX_dom -lFoX_sax -lFoX_wxml -lFoX_common -lFoX_utils -lFoX_fsys /apps/USE/easybuild/staging/2021.1/software/FFTW/3.3.9-nvompic-2021a/lib -lfftw3_omp -lfftw3 -L/apps/USE/easybuild/staging/2021.1/software/NVHPC/22.3/Linux_x86_64/22.3/compilers/lib -lblas
/apps/USE/easybuild/staging/2021.1/software/binutils/2.36.1-GCCcore-10.3.0/bin/ld: read in flex scanner failed
As you can see, there I reach another issue.
I need to search through the users mailing list to see if somebody else experienced something similar.
In case you have some suggestions, I'm happy to hear them.
Best regards,
Robert
-----Original Message-----
From: Paolo Giannozzi <paolo.giannozzi at uniud.it>
Sent: Monday, 23 May 2022 21:46
To: Quantum ESPRESSO users Forum <users at lists.quantum-espresso.org>; Robert MIJAKOVIC <robert.mijakovic at lxp.lu>
Subject: Re: [QE-users] [QE 7.0/GPU] Build configured with Autotools fails
Here some explanations:
https://www.mail-archive.com/users@lists.quantum-espresso.org/msg41494.html
https://www.mail-archive.com/users@lists.quantum-espresso.org/msg40215.html
The path to the fftw3.f03 file, needed by fftw3, must be specified in INCLUDE_FFTW or similar variables. The latest development version (soon to be released) has somewhat better FFTW3 auto-detection
Paolo
On 23/05/2022 17:55, Robert MIJAKOVIC wrote:
> # Summary
>
> QE 7.0/GPU compilation with Autotools fails with "Can't find include
> file fftw3.f03 (fft_scalar.FFTW3.f90: 40)"
>
> # Version
>
> qe-7.0-ReleasePack.tgz
>
> # Environment
>
> ## Hardware
>
> 1. 2xAMD EPYC 7452
>
> 2. 4xNVIDIA A100
>
> 3. 512 GB RAM
>
> ## Software
>
> 1. OS: Rocky Linux release 8.5 (Green Obsidian)
>
> 2. NVHPC 22.3
>
> 3. OpenMPI 4.1.3 built with NVHPC 22.3
>
> 4. CUDA 11.3.1 with Driver 470.82.01
>
> 5. libxc 5.1.5
>
> 6. Autotools 20210128
>
> # Steps to reproduce
>
> ## Configured with:
>
> `./configure --prefix=$HOME/install/qe-7.0 --enable-openmp
> --with-scalapack=no --enable-cuda-env-check=no
> --with-cuda=$EBROOTNVHPC/Linux_x86_64/22.3/cuda/11.0
> --with-cuda-runtime=11.0 --with-cuda-cc=80`
>
> ## Prebuild options
>
> `cp $EBROOTLIBXC/include/*.mod Modules/mod/qe_modules && export
> FPP='nvfortran -Mpreprocess -E' && export CPP='cpp -E' && export
> FCPP='cpp -E' && `
>
> ## make options
>
> `make all epw`
>
> # Observed behavior
>
> Compilation fails with:
>
> ```
>
> make[1]: Entering directory
> '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib'
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_param.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_support.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> stick_base.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_types.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> scatter_mod.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_scatter_2d.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_buffers.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_scatter_gpu.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_scatter_2d_gpu.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_ggen.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_scalar.DFTI.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_scalar.ESSL.f90
>
> NVFORTRAN-W-0006-Input file empty (fft_scalar.ESSL.f90)
>
> NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fftw_interfaces.f90
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_scalar.FFTW.f90
>
> NVFORTRAN-W-0006-Input file empty (fft_scalar.FFTW.f90)
>
> NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings
>
> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI -cuda -gpu=cc80,cuda11.0
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude -c
> fft_scalar.FFTW3.f90
>
> NVFORTRAN-F-0906-Can't find include file fftw3.f03
> (fft_scalar.FFTW3.f90: 40)
>
> NVFORTRAN/x86-64 Linux 22.3-0: compilation aborted
>
> make[1]: *** [../make.inc:16: fft_scalar.FFTW3.o] Error 2
>
> make[1]: Leaving directory '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib'
>
> make: *** [Makefile:196: libfft] Error 1
>
> ```
>
> # Questions
>
> 1. What do I do wrong?
>
> 2. Why configure didn't fail if FFTW is mandatory?
>
> 3. Is it possible to configure the build system to use cuFFTinstead?
>
> Dr. rer. nat. Robert Mijaković | HPC System Software Architect
>
>
> *Lux**Provide*
> 3, Op der Poukewiss | L-7795 Bissen
>
> Grand-Duchy of Luxembourg
> M (+352) 691 396 474
>
> robert.mijakovic at lxp.lu
> <mailto:robert.mijakovic at lxp.lu> |www.luxprovide.lu
> <http://www.luxprovide.lu/>
>
>
> _______________________________________________
> The Quantum ESPRESSO community stands by the Ukrainian people and
> expresses its concerns about the devastating effects that the Russian
> military offensive has on their country and on the free and peaceful
> scientific, cultural, and economic cooperation amongst peoples
> _______________________________________________
> Quantum ESPRESSO is supported by MaX (www.max-centre.eu) users mailing
> list users at lists.quantum-espresso.org
> https://lists.quantum-espresso.org/mailman/listinfo/users
--
Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche, Univ. Udine, via delle Scienze 208, 33100 Udine, Italy Phone +39-0432-558216, fax +39-0432-558222
More information about the users
mailing list