[QE-users] [QE 7.0/GPU] Build configured with Autotools fails

[Paolo Giannozzi]

On 5/24/22 17:37, Robert MIJAKOVIC wrote:

> I tried to specify the location of fftw3.f03 file by setting INCLUDE_FFTW variable

should be "FFTW_INCLUDE"

> mpif90 -mp -cuda -gpu=cc80,cuda11.0 -acc -o pw.x \
>     pwscf.o  libpw.a ../../KS_Solvers/libks_solvers.a ../../dft-d3/libdftd3qe.a /mnt/tier2/users/apps/scratchpad/qe-7.0//Modules/libqemod.a /mnt/tier2/users/apps/scratchpad/qe-7.0//upflib/libupf.a /mnt/tier2/users/apps/scratchpad/qe-7.0//XClib/xc_lib.a /mnt/tier2/users/apps/scratchpad/qe-7.0//FFTXlib/libqefft.a /mnt/tier2/users/apps/scratchpad/qe-7.0//LAXlib/libqela.a /mnt/tier2/users/apps/scratchpad/qe-7.0//UtilXlib/libutil.a /mnt/tier2/users/apps/scratchpad/qe-7.0//MBD/libmbd.a  -cudalib=cufft,cublas,cusolver,curand /mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src/libdevXlib.a -L/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src -ldevXlib  -L/apps/USE/easybuild/staging/2021.1/software/NVHPC/22.3/Linux_x86_64/22.3/compilers/lib -llapack -L/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/lib  -lFoX_dom -lFoX_sax -lFoX_wxml -lFoX_common -lFoX_utils -lFoX_fsys  /apps/USE/easybuild/staging/2021.1/software/FFTW/3.3.9-nvompic-2021a/lib 

should be "-L/apps/..."

> /apps/USE/easybuild/staging/2021.1/software/binutils/2.36.1-GCCcore-10.3.0/bin/ld: read in flex scanner failed

maybe it is a consequence of the above point? if not, there is something 
(very) strange in your software

Paolo

> As you can see, there I reach another issue.
> I need to search through the users mailing list to see if somebody else experienced something similar.
> 
> In case you have some suggestions, I'm happy to hear them.
> 
> Best regards,
> Robert
> 
> -----Original Message-----
> From: Paolo Giannozzi <paolo.giannozzi at uniud.it>
> Sent: Monday, 23 May 2022 21:46
> To: Quantum ESPRESSO users Forum <users at lists.quantum-espresso.org>; Robert MIJAKOVIC <robert.mijakovic at lxp.lu>
> Subject: Re: [QE-users] [QE 7.0/GPU] Build configured with Autotools fails
> 
> Here some explanations:
> https://www.mail-archive.com/users@lists.quantum-espresso.org/msg41494.html
> https://www.mail-archive.com/users@lists.quantum-espresso.org/msg40215.html
> The path to the fftw3.f03 file, needed by fftw3, must be specified in INCLUDE_FFTW or similar variables. The latest development version (soon to be released) has somewhat better FFTW3 auto-detection
> 
> Paolo
> 
> On 23/05/2022 17:55, Robert MIJAKOVIC wrote:
>> # Summary
>>
>> QE 7.0/GPU compilation with Autotools fails with "Can't find include
>> file fftw3.f03 (fft_scalar.FFTW3.f90: 40)"
>>
>> # Version
>>
>> qe-7.0-ReleasePack.tgz
>>
>> # Environment
>>
>> ## Hardware
>>
>> 1. 2xAMD EPYC 7452
>>
>> 2. 4xNVIDIA A100
>>
>> 3. 512 GB RAM
>>
>> ## Software
>>
>> 1. OS: Rocky Linux release 8.5 (Green Obsidian)
>>
>> 2. NVHPC 22.3
>>
>> 3. OpenMPI 4.1.3 built with NVHPC 22.3
>>
>> 4. CUDA 11.3.1 with Driver 470.82.01
>>
>> 5. libxc 5.1.5
>>
>> 6. Autotools 20210128
>>
>> # Steps to reproduce
>>
>> ## Configured with:
>>
>> `./configure --prefix=$HOME/install/qe-7.0 --enable-openmp
>> --with-scalapack=no --enable-cuda-env-check=no
>> --with-cuda=$EBROOTNVHPC/Linux_x86_64/22.3/cuda/11.0
>> --with-cuda-runtime=11.0 --with-cuda-cc=80`
>>
>> ## Prebuild options
>>
>> `cp $EBROOTLIBXC/include/*.mod Modules/mod/qe_modules && export
>> FPP='nvfortran -Mpreprocess -E' && export CPP='cpp -E' && export
>> FCPP='cpp -E' && `
>>
>> ## make options
>>
>> `make all epw`
>>
>> # Observed behavior
>>
>> Compilation fails with:
>>
>> ```
>>
>> make[1]: Entering directory
>> '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib'
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_param.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_support.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> stick_base.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_types.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> scatter_mod.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_scatter_2d.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_buffers.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_scatter_gpu.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_scatter_2d_gpu.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_ggen.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_scalar.DFTI.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_scalar.ESSL.f90
>>
>> NVFORTRAN-W-0006-Input file empty (fft_scalar.ESSL.f90)
>>
>> NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fftw_interfaces.f90
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_scalar.FFTW.f90
>>
>> NVFORTRAN-W-0006-Input file empty (fft_scalar.FFTW.f90)
>>
>> NVFORTRAN/x86-64 Linux 22.3-0: compilation completed with warnings
>>
>> mpif90 -fast -Mcache_align -Mpreprocess -Mlarge_arrays -mp -D__PGI
>> -D__CUDA -D__USE_CUSOLVER -D__FFTW3 -D__MPI  -cuda -gpu=cc80,cuda11.0
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/include
>> -acc -I/mnt/tier2/users/apps/scratchpad/qe-7.0//external/devxlib/src
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//include
>> -I/mnt/tier2/users/apps/scratchpad/qe-7.0//FoX/finclude   -c
>> fft_scalar.FFTW3.f90
>>
>> NVFORTRAN-F-0906-Can't find include file fftw3.f03
>> (fft_scalar.FFTW3.f90: 40)
>>
>> NVFORTRAN/x86-64 Linux 22.3-0: compilation aborted
>>
>> make[1]: *** [../make.inc:16: fft_scalar.FFTW3.o] Error 2
>>
>> make[1]: Leaving directory '/mnt/tier2/users/apps/scratchpad/qe-7.0/FFTXlib'
>>
>> make: *** [Makefile:196: libfft] Error 1
>>
>> ```
>>
>> # Questions
>>
>> 1. What do I do wrong?
>>
>> 2. Why configure didn't fail if FFTW is mandatory?
>>
>> 3. Is it possible to configure the build system to use cuFFTinstead?
>>
>> Dr. rer. nat. Robert Mijaković | HPC System Software Architect
>>
>>
>> *Lux**Provide*
>> 3, Op der Poukewiss | L-7795 Bissen
>>
>> Grand-Duchy of Luxembourg
>> M (+352) 691 396 474
>>
>> robert.mijakovic at lxp.lu
>> <mailto:robert.mijakovic at lxp.lu> |www.luxprovide.lu
>> <http://www.luxprovide.lu/>
>>
>>
>> _______________________________________________
>> The Quantum ESPRESSO community stands by the Ukrainian people and
>> expresses its concerns about the devastating effects that the Russian
>> military offensive has on their country and on the free and peaceful
>> scientific, cultural, and economic cooperation amongst peoples
>> _______________________________________________
>> Quantum ESPRESSO is supported by MaX (www.max-centre.eu) users mailing
>> list users at lists.quantum-espresso.org
>> https://lists.quantum-espresso.org/mailman/listinfo/users
> 
> --
> Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche, Univ. Udine, via delle Scienze 208, 33100 Udine, Italy Phone +39-0432-558216, fax +39-0432-558222

-- 
Paolo Giannozzi, Dip. Scienze Matematiche Informatiche e Fisiche,
Univ. Udine, via delle Scienze 206, 33100 Udine, Italy
Phone +39-0432-558216, fax +39-0432-558222